IPv4/IPv6 Dual-Stack

Dual-stack allows pods and services to have both IPv4 and IPv6 addresses simultaneously. GA since Kubernetes 1.21, dual-stack is the foundation for IPv6-only clusters and for clusters that must serve both address families during a long migration.

What This Page Covers
  • Dual-stack GA history and what changed from alpha/beta
  • IPv6 address types: link-local, ULA (fc00::/7), global unicast; why not use link-local for pods
  • CIDR planning: pod CIDRs, service CIDRs, node CIDRs for both families
  • API server flags: --service-cluster-ip-range, kube-controller-manager --cluster-cidr and --node-cidr-mask-size
  • kubeadm dual-stack ClusterConfiguration YAML
  • kubelet nodeIP dual-stack configuration
  • Pod dual-stack: status.podIPs (both IPs), status.podIP (primary)
  • Service ipFamilyPolicy: SingleStack, PreferDualStack, RequireDualStack
  • Service ipFamilies: [IPv4, IPv6] vs [IPv6, IPv4] — primary family ordering
  • Service ClusterIPs: two entries; headless services; ExternalName services
  • Service NodePort dual-stack: same port on both IPv4 and IPv6 node addresses
  • LoadBalancer dual-stack: cloud provider support; AWS NLB, GCE, Azure
  • DNS: A records (IPv4) and AAAA records (IPv6); CoreDNS dual-stack behavior
  • EndpointSlices: separate slices per address family (addressType: IPv4 vs IPv6)
  • kube-proxy dual-stack: iptables + ip6tables; IPVS mode
  • CNI dual-stack support: Calico, Cilium, Flannel, AWS VPC CNI limitations
  • NetworkPolicy with IPv6 ipBlock CIDRs
  • Node dual-stack addresses: status.addresses; primary IP selection
  • Migration path: single-stack IPv4 → dual-stack → single-stack IPv6
  • IPv6-only clusters: feature gate, CNI requirements
  • Troubleshooting: AAAA record missing, pod IPv6 unreachable, service CIDR overlap
  • 6 metrics + 4 alerting rules + 5 troubleshooting runbooks

Why Dual-Stack

IPv4 address exhaustion is real. Many organizations are under pressure to adopt IPv6, but cannot flip a switch and drop IPv4 overnight — applications, firewalls, monitoring systems, and load balancers all need time to migrate. Dual-stack gives each pod and service addresses in both families, allowing gradual migration without a hard cutover.

Key use cases:

  • Regulatory or network mandate — some government and carrier networks require IPv6 reachability for all workloads.
  • IPv6-only node pools — new cloud instances get only IPv6; dual-stack gateway bridges to legacy IPv4 services.
  • Migration staging — run both families, shift traffic to IPv6, then drop IPv4.
  • Edge/IoT — IoT devices increasingly use IPv6-only networks.
ℹ️
Feature history

Dual-stack was alpha in 1.16 (opt-in feature gate, single-stack architecture), beta in 1.20 (feature gate on by default, architecture rework), and GA in 1.21 (feature gate removed, always available). Clusters created before 1.21 on alpha/beta dual-stack need verification — the beta implementation had breaking changes.

IPv6 Address Types Primer

Not all IPv6 addresses are suitable for pod or service CIDRs. Understanding the types prevents misconfiguration:

IPv4 ranges used in K8s

  • 10.0.0.0/8 — pod CIDRs (common)
  • 172.16.0.0/12 — pod or node CIDRs
  • 192.168.0.0/16 — small clusters
  • 10.96.0.0/12 — service CIDR (kubeadm default)
  • Must not overlap with node network or each other

IPv6 ranges for K8s

  • fc00::/7 — Unique Local Address (ULA); routable within org; recommended for pods/services
  • fd00::/8 — ULA with L=1 bit; use fd prefix for pods (e.g., fd00:10:244::/48)
  • 2000::/3 — Global Unicast (GUA); internet-routable; use for node addresses
  • fe80::/10 — Link-local; NOT routable across nodes; never use for pods
  • ::1/128 — loopback; equivalent of 127.0.0.1
⚠️
Link-local addresses (fe80::/10) cannot be used for pod or service CIDRs

Link-local addresses are scoped to a single network segment — they are not routable between nodes. If a CNI assigns link-local addresses to pods, cross-node pod communication will fail silently. Always use ULA (fd prefix) or GUA for pod and service CIDRs.

CIDR Planning for Dual-Stack

A dual-stack cluster requires four non-overlapping CIDR blocks — two per address family:

Address Family Role Example CIDR Notes ━━━━━━━━━━━━━━ ━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ IPv4 Pod CIDR 10.244.0.0/16 /24 per node (256 IPs) IPv4 Service CIDR 10.96.0.0/16 ClusterIPs allocated from here IPv6 Pod CIDR fd00:10:244::/48 /64 per node (2^64 IPs) IPv6 Service CIDR fd00:10:96::/112 /128 per service = 65536 services Nodes: each node gets a /24 (IPv4) and /64 (IPv6) subnet from the pod CIDRs e.g., node-1: 10.244.1.0/24 + fd00:10:244:1::/64 node-2: 10.244.2.0/24 + fd00:10:244:2::/64 Critical: these four CIDRs must not overlap with each other or with node networking
⚠️
IPv6 /112 for Services

The service CIDR for IPv6 should be at most /112 — this gives 65536 service IPs. Kubernetes allocates service IPs sequentially, so a very large IPv6 service CIDR doesn't help; the controller only allocates from the start of the range. A /112 matches typical IPv4 service CIDR sizing.

Cluster Configuration

API Server & Controller Manager Flags

Both CIDRs are passed as comma-separated pairs (IPv4 first is conventional but not required):

# kube-apiserver
--service-cluster-ip-range=10.96.0.0/16,fd00:10:96::/112

# kube-controller-manager
--cluster-cidr=10.244.0.0/16,fd00:10:244::/48
--service-cluster-ip-range=10.96.0.0/16,fd00:10:96::/112
--node-cidr-mask-size-ipv4=24     # each node gets a /24 from 10.244.0.0/16
--node-cidr-mask-size-ipv6=64     # each node gets a /64 from fd00:10:244::/48

# kube-proxy
--cluster-cidr=10.244.0.0/16,fd00:10:244::/48

kubeadm Dual-Stack ClusterConfiguration

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: "1.31.0"
networking:
  podSubnet: "10.244.0.0/16,fd00:10:244::/48"       # comma-separated dual-stack
  serviceSubnet: "10.96.0.0/16,fd00:10:96::/112"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.1.10"    # control plane node IPv4 address
nodeRegistration:
  kubeletExtraArgs:
    node-ip: "192.168.1.10,fd00::1"   # comma-separated: IPv4,IPv6 of this node

kubelet nodeIP — Dual-Stack Node Registration

Each kubelet must be told which IPs to use for the node. Without this, it picks a single IP automatically (may miss the IPv6 address):

# /var/lib/kubelet/kubeadm-flags.env  (or KubeletConfiguration)
KUBELET_KUBEADM_ARGS="--node-ip=192.168.1.11,fd00::2"

# Or in KubeletConfiguration:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeIP: "192.168.1.11,fd00::2"    # IPv4,IPv6 comma-separated

Pod Dual-Stack Addresses

When a pod is scheduled on a dual-stack node, the CNI assigns two IPs — one per address family. Both appear in status.podIPs:

kubectl get pod my-pod -o yaml
# status:
#   podIP: 10.244.1.5                   # primary IP (first family configured)
#   podIPs:
#     - ip: 10.244.1.5                  # IPv4
#     - ip: fd00:10:244:1::5            # IPv6
#   hostIP: 192.168.1.11
#   hostIPs:
#     - ip: 192.168.1.11
#     - ip: fd00::2

# Inside the pod — both interfaces visible:
kubectl exec my-pod -- ip addr show eth0
# inet  10.244.1.5/24 scope global eth0
# inet6 fd00:10:244:1::5/64 scope global eth0
# inet6 fe80::...  scope link eth0       ← link-local, always present, not routable
ℹ️
Primary IP and address family ordering

status.podIP (singular) is the primary IP — it matches the first family in the cluster's --cluster-cidr ordering. If the cluster was configured IPv4-first (10.244.0.0/16,fd00::/48), podIP is always IPv4. Applications that read only podIP (e.g., via the Downward API) will see only the primary family IP.

Using Pod IPs from Downward API

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP        # primary IP only (IPv4 if IPv4-first cluster)
  - name: POD_IPS
    valueFrom:
      fieldRef:
        fieldPath: status.podIPs       # returns comma-separated: "10.244.1.5,fd00:10:244:1::5"

Service Dual-Stack Configuration

ipFamilyPolicy

spec.ipFamilyPolicy controls how the Service allocates IPs across address families:

PolicyBehaviorClusterIPs allocated
SingleStackOnly one address family; default when cluster is single-stack1 (from preferred family)
PreferDualStackDual-stack if cluster supports it, single-stack otherwise2 on dual-stack cluster, 1 on single-stack
RequireDualStackMust have both families; creation fails on single-stack cluster2 (error if cluster is single-stack)

ipFamilies — Address Family Ordering

spec.ipFamilies specifies which families to use and in which order. The first family is the primary (its IP goes into spec.clusterIP):

# Dual-stack service — IPv4 primary
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: production
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
    - IPv4             # primary family → goes into spec.clusterIP
    - IPv6             # secondary family

# After creation:
# spec.clusterIP:  10.96.100.5             (IPv4, primary)
# spec.clusterIPs:
#   - 10.96.100.5                          (IPv4)
#   - fd00:10:96::100:5                    (IPv6)
# IPv6-primary dual-stack service
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
    - IPv6             # primary: spec.clusterIP will be IPv6
    - IPv4

# Single-stack IPv6 service on a dual-stack cluster
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv6             # only allocate IPv6 ClusterIP
ℹ️
Existing services on cluster upgrade to dual-stack

When a single-stack cluster is upgraded to dual-stack, existing Services retain their SingleStack policy. They are NOT automatically upgraded to dual-stack. To enable dual-stack on existing services, patch spec.ipFamilyPolicy: PreferDualStack — this causes a new ClusterIP to be allocated for the second family.

Headless Services

Headless services (clusterIP: None) with RequireDualStack return both A and AAAA records per pod:

apiVersion: v1
kind: Service
metadata:
  name: my-headless
  namespace: production
spec:
  clusterIP: None
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
    - IPv4
    - IPv6
  selector:
    app: my-app

# DNS result for my-headless.production.svc.cluster.local:
# A    records: one per pod IPv4 address
# AAAA records: one per pod IPv6 address

NodePort Services

A NodePort service on a dual-stack cluster listens on the same port on both the IPv4 and IPv6 node addresses:

# NodePort 30080 is reachable at:
# http://192.168.1.11:30080   (node IPv4)
# http://[fd00::2]:30080      (node IPv6)

# kube-proxy programs both iptables and ip6tables rules for NodePort services

LoadBalancer Services

Cloud provider support for dual-stack LoadBalancer services is provider-specific:

Cloud ProviderDual-Stack LB SupportNotes
AWS NLBYes (IP mode)Requires aws-load-balancer-ip-address-type: dualstack annotation; NLB only (not ALB)
GCE/GKEYesSet ipFamilyPolicy: RequireDualStack; GKE provisions dual-stack L4 LB automatically
Azure AKSYes (1.26+)Cluster must be provisioned as dual-stack; both families on same LB frontend
MetalLBYes (v0.13+)Configure separate IPv4 and IPv6 address pools; assign both to the service
# AWS NLB dual-stack LoadBalancer service
apiVersion: v1
kind: Service
metadata:
  name: my-nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: "dualstack"
spec:
  type: LoadBalancer
  ipFamilyPolicy: RequireDualStack
  ipFamilies: [IPv4, IPv6]
  ports:
    - port: 443
      targetPort: 8443
  selector:
    app: my-app

DNS — A and AAAA Records

CoreDNS automatically generates both A (IPv4) and AAAA (IPv6) records for dual-stack services. The kubernetes plugin watches the API server and populates both record types from spec.clusterIPs:

# Verify dual-stack DNS records for a service
kubectl exec -n default deploy/debug -- nslookup my-service.production.svc.cluster.local
# Server:    10.96.0.10
# Address 1: 10.96.100.5 my-service.production.svc.cluster.local    ← A record (IPv4)
# Address 2: fd00:10:96::100:5 my-service.production.svc.cluster.local ← AAAA record (IPv6)

# Explicit A record lookup
kubectl exec -n default deploy/debug -- dig A my-service.production.svc.cluster.local

# Explicit AAAA record lookup
kubectl exec -n default deploy/debug -- dig AAAA my-service.production.svc.cluster.local

# PTR records work for both families:
# IPv4: 5.100.96.10.in-addr.arpa → my-service.production.svc.cluster.local
# IPv6: reverse of fd00:10:96::100:5 → my-service.production.svc.cluster.local
ℹ️
Application address family selection

When a dual-stack service has both A and AAAA records, the client application chooses which to connect to. Most modern applications use Happy Eyeballs (RFC 8305) — they try IPv6 first and fall back to IPv4 if IPv6 doesn't connect within 250ms. Applications that do not support Happy Eyeballs (or that use AF_INET explicitly) will always use IPv4 even if AAAA records exist.

EndpointSlices — Per Address Family

kube-proxy and other consumers of endpoint data need to know which IP family each endpoint belongs to. Kubernetes creates separate EndpointSlice objects per address family for dual-stack services:

kubectl get endpointslices -n production -l kubernetes.io/service-name=my-service
# NAME                     ADDRESSTYPE   PORTS   ENDPOINTS          AGE
# my-service-abc12         IPv4          8080    10.244.1.5,...     5m
# my-service-def34         IPv6          8080    fd00:10:244:1::5,..  5m

# Inspect the IPv6 slice:
kubectl get endpointslice my-service-def34 -n production -o yaml
# spec:
#   addressType: IPv6
#   endpoints:
#     - addresses: ["fd00:10:244:1::5"]
#       conditions:
#         ready: true
#         serving: true
#         terminating: false

kube-proxy Dual-Stack

kube-proxy programs both iptables (IPv4) and ip6tables (IPv6) rule sets when running in dual-stack mode. The same Service gets two sets of rules — one per address family.

# Verify both rule sets are populated (iptables mode)
sudo iptables  -t nat -L KUBE-SERVICES | grep 10.96.100.5    # IPv4 ClusterIP rules
sudo ip6tables -t nat -L KUBE-SERVICES | grep "fd00:10:96"    # IPv6 ClusterIP rules

# IPVS mode: virtual servers for both families
sudo ipvsadm -Ln | grep -E "10\.96\.100\.5|fd00:10:96"

# KubeProxyConfiguration for dual-stack (no special config needed — auto-detected):
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "iptables"
clusterCIDR: "10.244.0.0/16,fd00:10:244::/48"    # comma-separated dual-stack

CNI Dual-Stack Support

Calico

  • Full dual-stack support
  • Set CALICO_IPV6POOL_CIDR in calico-node DaemonSet
  • BGP advertises both IPv4 and IPv6 prefixes
  • IPIP/VXLAN encapsulation for both families
  • IPPool CRD: create one per family

Cilium

  • Full dual-stack support
  • Helm: --set ipv6.enabled=true
  • eBPF programs handle both IPv4 and IPv6 natively
  • Can replace kube-proxy for both families
  • Hubble shows both address families in flows

Flannel

  • Limited dual-stack support (added 0.15+)
  • Requires EnableIPv6: true in net-conf.json
  • vxlan backend only for IPv6
  • No NetworkPolicy enforcement (requires Canal)
  • Not recommended for new dual-stack deployments

AWS VPC CNI

  • IPv6 mode is single-stack IPv6 only (not dual-stack)
  • Each pod gets a single IPv6 address from the VPC subnet
  • Requires instance types with IPv6 ENI support
  • IPv6 mode: ENABLE_IPv6=true, ENABLE_PREFIX_DELEGATION=true
  • Cannot run both IPv4 and IPv6 per pod simultaneously

Calico Dual-Stack Configuration

# In the calico-node DaemonSet environment:
- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
- name: CALICO_IPV6POOL_CIDR
  value: "fd00:10:244::/48"
- name: CALICO_IPV6POOL_IPIP
  value: "CrossSubnet"
- name: FELIX_IPV6SUPPORT
  value: "true"
- name: IP6
  value: "autodetect"                   # auto-detect node IPv6 address
- name: IP6_AUTODETECTION_METHOD
  value: "first-found"

# Create both IPPools (or let Calico create them automatically):
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv6-pool
spec:
  cidr: fd00:10:244::/48
  ipipMode: Never
  vxlanMode: CrossSubnet
  natOutgoing: false                   # no MASQUERADE for IPv6 (routable ULA)
  nodeSelector: all()

Cilium Dual-Stack Configuration

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --set ipv4.enabled=true \
  --set ipv6.enabled=true \
  --set ipam.mode=kubernetes \
  --set k8s.requireIPv4PodCIDR=true \
  --set k8s.requireIPv6PodCIDR=true

# Verify:
cilium status
# IPv4: ✅
# IPv6: ✅

NetworkPolicy with IPv6

Standard NetworkPolicy ipBlock works with IPv6 CIDRs. The syntax is identical — just use IPv6 CIDR notation:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ipv6-external
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    # Allow IPv4 external
    - to:
        - ipBlock:
            cidr: 203.0.113.0/24
      ports:
        - protocol: TCP
          port: 443
    # Allow IPv6 external
    - to:
        - ipBlock:
            cidr: 2001:db8::/32           # IPv6 CIDR
            except:
              - 2001:db8:bad::/48         # except known bad range
      ports:
        - protocol: TCP
          port: 443
    # DNS (both IPv4 and IPv6 paths to CoreDNS)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Node Dual-Stack Addresses

kubectl get node node-1 -o yaml
# status:
#   addresses:
#     - type: InternalIP
#       address: 192.168.1.11        # IPv4 InternalIP
#     - type: InternalIP
#       address: fd00::2             # IPv6 InternalIP
#     - type: Hostname
#       address: node-1

# The primary InternalIP is used for:
# - kubelet registration
# - kube-proxy routing
# - CNI overlay tunnel endpoints
# The ordering follows --node-ip flag (first = primary)

IPv6-Only Clusters

An IPv6-only cluster uses a single address family — IPv6. This is distinct from dual-stack; pods and services have only IPv6 addresses. Requirements:

  • All nodes must have IPv6 addresses (no IPv4 required)
  • CNI must support IPv6-only mode (Cilium, Calico, AWS VPC CNI IPv6 mode)
  • --service-cluster-ip-range is IPv6 only: fd00:10:96::/112
  • --cluster-cidr is IPv6 only: fd00:10:244::/48
  • CoreDNS generates only AAAA records for services
  • External connectivity for IPv4-only upstream services requires NAT64/DNS64
# kubeadm for IPv6-only cluster
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  podSubnet: "fd00:10:244::/48"
  serviceSubnet: "fd00:10:96::/112"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "fd00::1"          # IPv6 control plane address
nodeRegistration:
  kubeletExtraArgs:
    node-ip: "fd00::1"                  # only IPv6 node IP

Migration Path

There is no automatic migration from single-stack to dual-stack. The process requires planning because existing Services and pods keep their single-stack configuration until explicitly updated:

Phase 1: Prepare infrastructure • Ensure all nodes have IPv6 addresses (configure OS/cloud networking) • Upgrade CNI to dual-stack capable version with IPv6 config Phase 2: Reconfigure cluster components • Update kube-apiserver: --service-cluster-ip-range=10.96.0.0/16,fd00:10:96::/112 • Update kube-controller-manager: --cluster-cidr + --node-cidr-mask-size-ipv6 • Update kube-proxy: --cluster-cidr (dual-stack) • Update kubelet on each node: --node-ip=IPv4,IPv6 Phase 3: Migrate existing resources • New pods automatically get dual-stack IPs after CNI is updated • Existing Services: patch spec.ipFamilyPolicy: PreferDualStack • New Services: create with RequireDualStack if needed Phase 4: Verify and test • Confirm pods have two IPs: kubectl get pod -o wide • Confirm services have two ClusterIPs: kubectl get svc -o yaml • Test IPv6 connectivity between pods • Test DNS AAAA records: dig AAAA svc.namespace.svc.cluster.local Phase 5 (optional): IPv6-only • Migrate all Services to SingleStack IPv6 • Disable IPv4 pod CIDR • Migrate to IPv6-only CNI mode
🚨
Rolling back dual-stack is not supported

Once a cluster is converted to dual-stack and services have been allocated IPv6 ClusterIPs, reverting to single-stack is not supported without cluster recreation. Services that received a second ClusterIP cannot have it removed while the service exists. Plan the migration carefully and test in a non-production cluster first.

Metrics, Alerting & Troubleshooting

Key Metrics

MetricSourceWhat It Tells You
kube_service_info{ip_family="IPv6"}kube-state-metricsCount of services with IPv6 ClusterIP allocated
kube_pod_info{pod_ip=~"fd.*"}kube-state-metricsPods with IPv6 primary IP (prefix match on fd)
coredns_dns_response_rcode_total{rcode="NXDOMAIN",qtype="AAAA"}CoreDNSFailed AAAA lookups; high rate = missing IPv6 ClusterIP
cilium_forward_count_total{family="6"}CiliumIPv6 packets forwarded by Cilium
felix_ipv6_supportCalico FelixWhether Felix has IPv6 support enabled; 0 = IPv6 will not work
kube_proxy_network_programming_duration_secondskube-proxyTime to program ip6tables rules; high = ip6tables contention

Alerting Rules

# Alert: Pod missing IPv6 address in dual-stack cluster
- alert: PodMissingIPv6Address
  expr: |
    count by (namespace, pod) (
      kube_pod_info{pod_ip!~"fd.*|2[0-9a-f]{3}:.*"}
    ) > 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} has no IPv6 address"

# Alert: Service missing IPv6 ClusterIP
- alert: ServiceMissingIPv6ClusterIP
  expr: |
    kube_service_spec_type{type="ClusterIP"} unless
    kube_service_info{ip_family="IPv6"}
  for: 15m
  labels:
    severity: info

# Alert: High AAAA NXDOMAIN rate
- alert: HighAAAANXDOMAINRate
  expr: |
    rate(coredns_dns_response_rcode_total{rcode="NXDOMAIN",qtype="AAAA"}[5m]) > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High rate of AAAA NXDOMAIN responses — services may lack IPv6 ClusterIPs"

# Alert: ip6tables rule programming failure
- alert: Ip6tablesRuleProgrammingFailure
  expr: rate(kube_proxy_sync_proxy_rules_errors_total[5m]) > 0
  for: 5m
  labels:
    severity: critical

Troubleshooting Runbooks

Runbook 1: Pod Has No IPv6 Address

# Check if pod has two IPs
kubectl get pod my-pod -o jsonpath='{.status.podIPs}'
# Expected: [{"ip":"10.244.1.5"},{"ip":"fd00:10:244:1::5"}]
# If only IPv4: CNI not configured for dual-stack

# Check if node has dual-stack pod CIDR
kubectl get node my-node -o jsonpath='{.spec.podCIDRs}'
# Expected: ["10.244.1.0/24","fd00:10:244:1::/64"]
# If only one: kube-controller-manager --cluster-cidr not set to dual-stack

# Check CNI configuration on node
cat /etc/cni/net.d/10-calico.conflist | grep -i ipv6
# Or for Cilium:
cilium status | grep IPv6

Runbook 2: Service Has No IPv6 ClusterIP

# Check service IP families
kubectl get svc my-service -o jsonpath='{.spec.ipFamilies} {.spec.ipFamilyPolicy}'
# If: [IPv4] SingleStack → not dual-stack configured

# Patch existing service to dual-stack
kubectl patch svc my-service -p '{"spec":{"ipFamilyPolicy":"PreferDualStack","ipFamilies":["IPv4","IPv6"]}}'

# Verify after patch
kubectl get svc my-service -o yaml | grep -A5 clusterIPs

# Check API server service CIDR includes IPv6
kubectl get cm -n kube-system kubeadm-config -o yaml | grep serviceSubnet

Runbook 3: AAAA Record Not Resolving

# Test from inside a pod
kubectl exec -n default deploy/debug -- dig AAAA my-service.production.svc.cluster.local

# If NXDOMAIN: service has no IPv6 ClusterIP
kubectl get svc my-service -n production -o jsonpath='{.spec.clusterIPs}'

# If SERVFAIL: CoreDNS issue
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 | grep -i error

# Check CoreDNS can reach API server for IPv6 endpoint data
kubectl exec -n kube-system deploy/coredns -- wget -qO- http://localhost:8080/health

Runbook 4: IPv6 Pod-to-Pod Communication Failing

# Verify IPv6 is reachable within node first
kubectl exec pod-a -- ping6 fd00:10:244:1::6   # same-node pod

# Cross-node IPv6 ping
kubectl exec pod-a -- ping6 fd00:10:244:2::5   # different-node pod

# If same-node works but cross-node fails: CNI overlay not configured for IPv6
# Calico: check FELIX_IPV6SUPPORT=true in calico-node DaemonSet
kubectl get ds calico-node -n kube-system -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="FELIX_IPV6SUPPORT")'

# Cilium: check IPv6 is enabled
cilium config view | grep ipv6

# Check node IPv6 routing table
kubectl exec -n kube-system calico-node-xxxx -- ip -6 route show

Runbook 5: CIDR Overlap Detected After Enabling Dual-Stack

# Check for CIDR overlaps
kubectl get cm -n kube-system kubeadm-config -o yaml

# Overlapping CIDRs cause pod-to-service routing to break silently
# Common mistake: node network uses fd00::/8 and pod CIDR also uses fd00::/48

# Verify all four CIDRs are disjoint:
# 1. Node IPv4 network (check cloud VPC / router)
# 2. Pod IPv4 CIDR     (kubectl get cm kubeadm-config)
# 3. Service IPv4 CIDR (kubectl get cm kubeadm-config)
# 4. Pod IPv6 CIDR     (must not overlap node IPv6 range)
# 5. Service IPv6 CIDR (must not overlap pod IPv6 range)

# If overlap exists: requires cluster rebuild — no in-place fix for CIDR conflicts

Best Practices

  1. Use ULA (fd prefix) for pod and service CIDRs — globally unique but not internet-routable; predictable; easy to distinguish from node addresses. Avoid fc (L=0) — use fd (L=1).
  2. Allocate /48 for pod IPv6 CIDR — this gives 65536 nodes each with a /64 (18 quintillion pod IPs per node). A /56 gives 256 nodes. Size based on your max node count.
  3. Use /112 for IPv6 service CIDR — 65536 services; matches typical IPv4 /16 service CIDR sizing. Larger ranges waste space without benefit.
  4. Enable dual-stack from day zero — retrofitting an existing single-stack cluster is painful and irreversible. If in doubt, enable it early even if you don't use IPv6 yet.
  5. Test IPv6 connectivity explicitly in CI — many integration tests only check IPv4. Add explicit IPv6 connectivity checks to your test suite.
  6. Configure --node-ip on all kubelets — without explicit node-ip, kubelet may pick the wrong IPv6 address (e.g., link-local) as the node's IPv6 address.
  7. Verify NetworkPolicy covers both address families — ipBlock rules must include both IPv4 and IPv6 CIDRs for complete coverage. A policy allowing only 10.0.0.0/8 does not block IPv6 connections.
  8. Set ipFamilyPolicy: RequireDualStack for services that must be reachable via IPv6PreferDualStack silently degrades to single-stack on single-stack clusters; RequireDualStack fails fast.