Windows Worker Nodes

Windows worker nodes allow Kubernetes clusters to run Windows Server containers alongside Linux workloads in the same cluster. The control plane always runs on Linux; only worker nodes can be Windows. This is a stable, production-supported configuration since Kubernetes 1.14 (GA), with significant improvements through 1.28+. Windows support targets workloads that cannot be containerized on Linux: legacy .NET Framework applications, COM+ services, Windows-specific drivers, and IIS-based services.

Mixed-OS Cluster Architecture

OS and Version Compatibility

Windows container image compatibility is strictly tied to the host OS build version. A container built on Windows Server 2022 cannot run on a Windows Server 2019 host, and vice versa — unless Hyper-V isolation is used (process isolation is default).

Windows Node Component Stack

Container Isolation Modes

# RuntimeClass for Hyper-V isolation
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: windows-hyperv
handler: runhcs-wcow-hypervisor   # containerd handler for Hyper-V
---
# Pod using Hyper-V isolation
spec:
  runtimeClassName: windows-hyperv
  nodeSelector:
    kubernetes.io/os: windows
  containers:
  - name: app
    image: mcr.microsoft.com/windows/servercore:ltsc2022

Scheduling Windows Pods

Because Linux and Windows nodes coexist in the same cluster, you must explicitly constrain Windows pods to Windows nodes. The scheduler does not automatically enforce OS compatibility.

# Minimum required: nodeSelector
spec:
  nodeSelector:
    kubernetes.io/os: windows

# Recommended: nodeSelector + toleration for Windows-specific taints
spec:
  nodeSelector:
    kubernetes.io/os: windows
    # optionally pin to specific Windows version:
    node.kubernetes.io/windows-build: "10.0.20348"   # WS2022

  # If you taint your Windows nodes (recommended):
  tolerations:
  - key: "os"
    value: "windows"
    operator: Equal
    effect: NoSchedule

# Prevent Linux pods from accidentally landing on Windows nodes:
# Add a taint to all Windows nodes:
kubectl taint node windows-node-1 os=windows:NoSchedule

# Linux pods don't tolerate this taint → never scheduled to Windows
# Windows pods add the toleration explicitly → works correctly

OS Version Scheduling Constraints

# Kubernetes adds node labels for Windows build version:
# node.kubernetes.io/windows-build: "10.0.17763"  (WS 2019)
# node.kubernetes.io/windows-build: "10.0.20348"  (WS 2022)
# node.kubernetes.io/windows-build: "10.0.26100"  (WS 2025)

# Pod that requires Windows Server 2022 specifically:
spec:
  nodeSelector:
    kubernetes.io/os: windows
    node.kubernetes.io/windows-build: "10.0.20348"

# Multi-arch image with OS version annotations (preferred):
# Use docker manifest or buildx to create a manifest list covering:
# - linux/amd64
# - windows/amd64:10.0.20348
# - windows/amd64:10.0.17763
# The runtime selects the correct layer automatically

Windows Networking

Windows container networking is fundamentally different from Linux. Instead of Linux network namespaces and iptables, Windows uses:

Windows CNI Network Modes

kube-proxy on Windows

# kube-proxy configuration for Windows (kernelspace mode)
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "kernelspace"              # only supported mode on Windows
winkernel:
  enableDSR: false               # Direct Server Return (requires WS 2019+ and overlay)
  networkName: "Calico"          # HNS network name (must match CNI network)
  sourceVip: "10.96.0.0"         # Required for overlay mode DSR
  rootHnsEndpointName: ""

# Note: iptables and ipvs modes are NOT supported on Windows
# kernelspace mode programs VFP rules for ClusterIP, NodePort, LoadBalancer

Resource Management on Windows

Windows resource isolation uses fundamentally different primitives than Linux cgroups. The Kubernetes API is the same, but the underlying enforcement differs:

# Windows pod resource spec (same API as Linux)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "1Gi"

# Windows-specific: no hugepages, no CPU manager static policy
# No cgroup v2 (Windows uses HCS/Job Objects)
# No NUMA-aware Memory Manager on Windows

Feature Parity: Linux vs Windows

HostProcess Containers

HostProcess containers (GA in Kubernetes 1.26) are the Windows equivalent of privileged containers on Linux. They run directly in the host network namespace and can access the node's filesystem and Windows API — used for DaemonSet-based node configuration, log collection, and network plugin management.

# HostProcess pod example (DaemonSet for node configuration)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: windows-node-agent
spec:
  selector:
    matchLabels:
      app: windows-node-agent
  template:
    metadata:
      labels:
        app: windows-node-agent
    spec:
      securityContext:
        windowsOptions:
          hostProcess: true          # enables HostProcess mode
          runAsUserName: "NT AUTHORITY\\SYSTEM"  # required for HostProcess
      hostNetwork: true              # required for HostProcess
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
      - key: "os"
        operator: Equal
        value: "windows"
        effect: NoSchedule
      containers:
      - name: node-agent
        image: myregistry/windows-agent:ltsc2022
        command: ["powershell.exe", "-Command", "C:\\agent\\start.ps1"]

Group Managed Service Accounts (GMSA)

GMSA allows Windows containers to authenticate to Active Directory resources (SQL Server, file shares, APIs) using machine identity — without storing passwords in Kubernetes Secrets.

# Step 1: Install the GMSA webhook (handles credential spec injection)
# https://github.com/kubernetes-sigs/windows-gmsa

# Step 2: Create GMSACredentialSpec CRD object
apiVersion: windows.k8s.io/v1
kind: GMSACredentialSpec
metadata:
  name: webapp-gmsa
spec:
  credspec:
    ActiveDirectoryConfig:
      GroupManagedServiceAccounts:
      - Name: webapp-svc-account
        Scope: CONTOSO
    CmsPlugins:
    - PluginGUID: "{DABB1965-CB8D-40B4-B527-F78BCFFBD657}"
    DomainJoinConfig:
      DnsName: contoso.com
      DnsTreeName: contoso.com
      Guid: "{guid-of-domain}"
      MachineAccountName: webapp-svc-account
      NetBiosName: CONTOSO
      Sid: "S-1-5-21-..."

# Step 3: ClusterRole + RoleBinding to allow service accounts to use the spec
# Step 4: Pod spec referencing the GMSA
spec:
  securityContext:
    windowsOptions:
      gmsaCredentialSpecName: webapp-gmsa
  nodeSelector:
    kubernetes.io/os: windows

Windows Node Setup

Prerequisites

# Windows Server requirements:
# - Windows Server 2019 LTSC (build 17763) or 2022 LTSC (build 20348)
# - Containers feature enabled
# - Hyper-V role (only if using Hyper-V isolation)
# - At least 4 CPUs, 8 GB RAM recommended

# Enable Containers feature
Install-WindowsFeature -Name containers -Restart

# (Optional) Enable Hyper-V for Hyper-V isolation
Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart

Install containerd on Windows

# Download and install containerd for Windows
$VERSION = "1.7.13"
curl.exe -L "https://github.com/containerd/containerd/releases/download/v${VERSION}/containerd-${VERSION}-windows-amd64.tar.gz" `
  -o containerd.tar.gz
tar -xvf containerd.tar.gz -C "C:\Program Files\containerd" --strip-components=1

# Create containerd service
containerd.exe config default | Out-File -Encoding ASCII C:\Windows\system32\config\systemprofile\AppData\Local\containerd\config.toml

# Register as Windows service
containerd.exe --register-service
Start-Service containerd

# Verify
Get-Service containerd
ctr.exe version

Install kubelet on Windows

# Download kubelet binary
$K8S_VERSION = "v1.29.2"
curl.exe -L "https://dl.k8s.io/release/${K8S_VERSION}/bin/windows/amd64/kubelet.exe" `
  -o "C:\k\kubelet.exe"

# kubelet config for Windows
# C:\k\kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  webhook:
    enabled: true
authorization:
  mode: Webhook
cgroupDriver: ""          # Not applicable on Windows
containerRuntimeEndpoint: "npipe:////./pipe/containerd-containerd"
clusterDNS:
- "10.96.0.10"
clusterDomain: "cluster.local"
rotateCertificates: true

# Start kubelet as service (typically via nssm or a startup script)
# kubelet.exe --kubeconfig=C:\k\kubeconfig --config=C:\k\kubelet-config.yaml `
#   --hostname-override=windows-node-1 `
#   --node-labels=kubernetes.io/os=windows

Install kube-proxy on Windows

# Download kube-proxy
curl.exe -L "https://dl.k8s.io/release/${K8S_VERSION}/bin/windows/amd64/kube-proxy.exe" `
  -o "C:\k\kube-proxy.exe"

# kube-proxy config
# C:\k\kube-proxy-config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "kernelspace"
clusterCIDR: "10.244.0.0/16"
winkernel:
  enableDSR: false
  networkName: "Calico"    # match your CNI network name

# Start: kube-proxy.exe --config=C:\k\kube-proxy-config.yaml

Storage on Windows

# Windows volume mounts use Windows paths
spec:
  containers:
  - name: app
    image: mcr.microsoft.com/windows/servercore:ltsc2022
    volumeMounts:
    - name: data
      mountPath: "C:\\data"         # Windows path; forward slashes also work: C:/data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

# Supported volume types on Windows:
# - emptyDir (tmpfs not supported; uses NTFS temp dir)
# - configMap / secret (projected as files)
# - persistentVolumeClaim (via Windows-compatible CSI drivers)
# - hostPath (Windows paths: C:\..., D:\...)

# Azure Disk CSI on Windows:
# storageClassName: managed-csi
# accessMode: ReadWriteOnce (block device, Windows NTFS formatted)

# Azure File CSI on Windows (SMB):
# storageClassName: azurefile-csi
# accessMode: ReadWriteMany (SMB share, mounted as network drive)

# SMB volume (direct, without CSI):
spec:
  volumes:
  - name: smb-share
    csi:
      driver: smb.csi.k8s.io
      readOnly: false
      volumeAttributes:
        source: "//smb-server.domain.com/share"
      nodeStageSecretRef:
        name: smb-creds
        namespace: default

Logging on Windows

# Container logs: same path structure as Linux
# /var/log/pods/__//.log
# On Windows: C:\var\log\pods\...

# Windows Event Log integration (for apps writing to Event Log):
# Use a sidecar or log agent that reads Windows Event Log
# e.g., Fluent Bit Windows build with winevtlog input plugin

# Fluent Bit Windows DaemonSet configuration (simplified):
[INPUT]
  Name         winevtlog
  Channels     Application,System,Security
  Interval_Sec 1
  DB           C:\fluent-bit\winevtlog.db

[OUTPUT]
  Name         es
  Match        *
  Host         elasticsearch.logging.svc.cluster.local

# Application stdout/stderr still captured by kubelet via CRI log path
# Windows applications must write to stdout/stderr for kubectl logs to work
# Use -RedirectStandardOutput / Console.WriteLine / Console.Error.WriteLine

Windows Node Diagnostics

# Check kubelet service status (PowerShell)
Get-Service kubelet
Get-EventLog -LogName Application -Source kubelet -Newest 50

# Check containerd
Get-Service containerd
ctr.exe -n k8s.io containers ls
ctr.exe -n k8s.io images ls

# Check kube-proxy
Get-Service kube-proxy

# HNS state (equivalent to ip netns / iptables on Linux):
Get-HNSNetwork                        # list HNS virtual networks
Get-HNSEndpoint                       # list container endpoints (pod IPs)
(Get-HNSEndpoint).IPAddress           # all pod IPs on this node

# VFP rules (kube-proxy equivalent of iptables-save):
vfpctrl.exe /list-vmswitch-port       # list switch ports
vfpctrl.exe /port  /list-rule   # Service rules on a port

# Check Windows node object
kubectl get node windows-node-1 -o yaml
kubectl describe node windows-node-1

# Run a diagnostic command inside a Windows container
kubectl exec -it my-windows-pod -- cmd.exe
kubectl exec -it my-windows-pod -- powershell.exe

# Windows container event logs via kubectl events
kubectl get events --field-selector involvedObject.name=my-windows-pod

Troubleshooting Runbooks

# Symptom: pod event "the container operating system does not match the host"

# 1. Get node Windows build
kubectl get node windows-node-1 -o jsonpath='{.metadata.labels.node\.kubernetes\.io/windows-build}'
# e.g.: 10.0.20348  (Windows Server 2022)

# 2. Get container image OS version
docker manifest inspect mcr.microsoft.com/windows/servercore:ltsc2019 | \
  jq '.manifests[] | select(.platform.os == "windows") | .platform'
# Look for "os.version": "10.0.17763.xxx"  (WS2019 = 17763)
# WS2022 = 20348; they don't match → failure

# 3. Fix: use the correct image tag for your host:
# ltsc2019 → Windows Server 2019 (17763)
# ltsc2022 → Windows Server 2022 (20348)
# Or use a multi-arch manifest that covers both

# 4. Add node build constraint to pod spec:
spec:
  nodeSelector:
    kubernetes.io/os: windows
    node.kubernetes.io/windows-build: "10.0.20348"

# Symptom: pod has no IP; kubectl get pod shows no IP; kubectl exec fails

# 1. Check kubelet logs for CNI errors
Get-EventLog -LogName Application -Source kubelet -Newest 100 | \
  Where-Object {$_.Message -like "*CNI*" -or $_.Message -like "*network*"}

# 2. Check HNS networks exist
Get-HNSNetwork | Select-Object Name, Type, AddressPrefix
# Expected: network matching your CNI config (e.g., "Calico" or "Flannel")

# 3. Check HNS endpoints
Get-HNSEndpoint | Where-Object {$_.VirtualNetwork -eq (Get-HNSNetwork -Name "Calico").ID}

# 4. Check CNI binary and config
ls C:\opt\cni\bin\
ls C:\etc\cni\net.d\

# 5. Restart CNI DaemonSet pod on the Windows node
kubectl delete pod -n kube-system -l k8s-app=calico-node \
  --field-selector spec.nodeName=windows-node-1

# 6. Check Windows Firewall rules (may be blocking pod traffic)
netsh advfirewall firewall show rule name=all | Select-String "Kubernetes"

# Symptom: Services unreachable from Windows pods

# 1. Check kube-proxy logs
Get-EventLog -LogName Application -Source "kube-proxy" -Newest 50

# 2. Verify HNS network name matches config
Get-HNSNetwork | Select-Object Name
# Compare to winkernel.networkName in kube-proxy config

# 3. Check VFP rules for a ClusterIP service
$svcIP = "10.96.0.100"   # ClusterIP of the service
Get-HNSEndpoint | Where-Object {$_.IPAddress -eq $svcIP}

# 4. Restart kube-proxy
Restart-Service kube-proxy

# 5. Check Windows Server version supports kernelspace mode
# Requires Windows Server 2019 1809+
[System.Environment]::OSVersion.Version

# Symptom: kubelet exits with "failed to connect to containerd"

# 1. Check containerd is running
Get-Service containerd

# 2. Verify the named pipe
[System.IO.Directory]::GetFiles("\\.\pipe\") | Where-Object {$_ -like "*containerd*"}
# Expected: \\.\pipe\containerd-containerd

# 3. Check kubelet containerRuntimeEndpoint config
# Should be: "npipe:////./pipe/containerd-containerd"
# NOT: "unix:///..." (Linux path) — common copy-paste error

# 4. Check containerd config for Windows
cat C:\Windows\system32\config\systemprofile\AppData\Local\containerd\config.toml
# Verify: address = "\\\\.\\pipe\\containerd-containerd"

# 5. Restart containerd then kubelet
Restart-Service containerd
Restart-Service kubelet

Production Best Practices

1. Always taint Windows nodes — add os=windows:NoSchedule to every Windows node. This prevents Linux pods (which don't have the toleration) from landing on Windows nodes, where they'd fail to start.
2. Pin pods to specific Windows build versions — use node.kubernetes.io/windows-build in nodeSelector or nodeAffinity. Process isolation requires exact build match; shipping a multi-arch manifest list is the cleanest long-term solution.
3. Use containerd 1.7+ on Windows — earlier versions have known CRI compatibility issues with Kubernetes 1.26+ (v1alpha2 removal). Verify with ctr.exe version.
4. Use Windows LTSC releases only — Semi-Annual Channel (SAC) releases (1903, 20H2, etc.) are EOL. Use LTSC 2019 or LTSC 2022. LTSC 2025 is available from Kubernetes 1.32+.
5. Test graceful termination carefully — Windows processes receive a console control event (not SIGTERM). Ensure your .NET/Windows applications handle AppDomain.CurrentDomain.ProcessExit or Console.CancelKeyPress. Many Windows services do not handle termination gracefully by default.
6. Use HostProcess containers for node management DaemonSets — instead of running node configuration scripts directly on the VM, use HostProcess pods to deploy CNI plugins, log agents, and monitoring tools. This gives you the same GitOps/lifecycle benefits as Linux DaemonSets.
7. Validate HNS state after kubelet restarts — kubelet restart on Windows can leave stale HNS endpoints. Monitor for pods that have IPs but no HNS endpoint, and add a node-level health check that recycles the CNI DaemonSet pod when this occurs.
8. Windows nodes have higher base memory overhead — Windows Server Core uses ~2 GB RAM at idle before any workload. Size Windows nodes with at least 8 GB RAM minimum; 16 GB+ for production. Adjust kubeReserved accordingly (recommend 2 Gi minimum for memory).
9. Monitor Windows Event Log alongside kubectl logs — application failures often appear in Windows Event Log before they surface in container stdout. Deploy a Fluent Bit or similar agent with winevtlog input as a HostProcess DaemonSet.
10. Upgrade Windows nodes in a dedicated node pool — Windows node upgrades require node drain + OS upgrade + rejoin (or blue/green node replacement). Never upgrade in-place by patching the OS while the node serves workloads. Use cluster-autoscaler with a separate Windows node group or Cluster API MachineDeployments.