cloud-controller-manager

Why cloud-controller-manager Exists

Before Kubernetes 1.6, cloud-specific logic (AWS ELB creation, GCE route management, Azure VM metadata) was compiled directly into kube-apiserver, kube-controller-manager, and kubelet. This created a tight coupling: every cloud provider had to submit code to the upstream Kubernetes repo, and a bug in the AWS provider could break GKE clusters.

CCM extracts all cloud-specific code into a separate binary that runs alongside — not inside — the core Kubernetes components. This enables:

• Cloud providers to release fixes independently of the Kubernetes release cycle
• Clusters to run without any cloud provider (bare-metal, on-premises)
• Third-party cloud providers (Hetzner, DigitalOcean, OVH) to integrate without upstream changes
• The core Kubernetes binary to shrink and stabilize

CCM Architecture and the Cloud Provider Interface

CCM implements the same reconciliation loop pattern as kube-controller-manager (see 04-kube-controller-manager.html § Reconciliation Loop). It watches Kubernetes API objects and calls cloud provider APIs to converge state.

The Cloud Provider Interface (Go)

Every CCM implementation must implement the cloudprovider.Interface Go interface from k8s.io/cloud-provider. This defines the contract between Kubernetes and any cloud platform.

// k8s.io/cloud-provider/cloud.go (simplified)
type Interface interface {
    Initialize(clientBuilder ControllerClientBuilder, stop <-chan struct{})
    LoadBalancer() (LoadBalancer, bool)      // nil, false if not supported
    Instances() (Instances, bool)            // VM metadata lookup
    InstancesV2() (InstancesV2, bool)        // newer V2 interface
    Zones() (Zones, bool)                    // zone/region metadata
    Clusters() (Clusters, bool)              // cluster management (rarely used)
    Routes() (Routes, bool)                  // VPC route table management
    ProviderName() string                    // "aws", "gce", "azure", etc.
    HasClusterID() bool
}

type LoadBalancer interface {
    GetLoadBalancer(ctx, clusterName, service) (*LoadBalancerStatus, bool, error)
    GetLoadBalancerName(ctx, clusterName, service) string
    EnsureLoadBalancer(ctx, clusterName, service, nodes) (*LoadBalancerStatus, error)
    UpdateLoadBalancer(ctx, clusterName, service, nodes) error
    EnsureLoadBalancerDeleted(ctx, clusterName, service) error
}

type Instances interface {
    NodeAddresses(ctx, nodeName) ([]NodeAddress, error)
    NodeAddressesByProviderID(ctx, providerID) ([]NodeAddress, error)
    InstanceID(ctx, nodeName) (string, error)
    InstanceType(ctx, nodeName) (string, error)
    InstanceTypeByProviderID(ctx, providerID) (string, error)
    InstanceExistsByProviderID(ctx, providerID) (bool, error)
    InstanceShutdownByProviderID(ctx, providerID) (bool, error)
}

The Four Built-in Controllers

Node Controller

The Node Controller populates cloud-specific metadata on Node objects after they register. When a new Node appears with spec.providerID set but no zone/address annotations, this controller calls the cloud API to fetch instance details and patches the Node.

Fields populated by the Node Controller:

# After CCM processes the new node:
status:
  addresses:
  - type: InternalIP
    address: "10.0.1.42"
  - type: ExternalIP
    address: "54.123.45.67"     # AWS: EIP or public IP
  - type: InternalDNS
    address: "ip-10-0-1-42.ec2.internal"
  - type: ExternalDNS
    address: "ec2-54-123-45-67.compute-1.amazonaws.com"
  - type: Hostname
    address: "ip-10-0-1-42.ec2.internal"

# Labels added by cloud provider
labels:
  topology.kubernetes.io/zone: "us-east-1a"
  topology.kubernetes.io/region: "us-east-1"
  node.kubernetes.io/instance-type: "m5.xlarge"
  failure-domain.beta.kubernetes.io/zone: "us-east-1a"   # legacy label
  failure-domain.beta.kubernetes.io/region: "us-east-1"  # legacy label

# ProviderID set by kubelet, used by CCM to look up instance
spec:
  providerID: "aws:///us-east-1a/i-0abc123def456"

Route Controller

Kubernetes requires every Pod to be directly reachable from every other Pod without NAT. On cloud platforms, this means each node's pod CIDR must be programmed into the cloud's VPC routing table so that inter-node pod traffic is routed correctly.

# Example VPC route entry created by CCM (AWS)
# Destination: 10.244.1.0/24 (pod CIDR for node worker-1)
# Target:      i-0abc123def456 (EC2 instance ID for worker-1)

# This allows pod-on-worker-2 (10.244.2.5) to reach pod-on-worker-1 (10.244.1.3)
# without SNAT — traffic routes directly through the VPC.

# Check routes created by CCM
aws ec2 describe-route-tables --filter Name=tag:kubernetes.io/cluster/my-cluster,Values=owned \
  --query 'RouteTables[].Routes[]' | jq '.[] | select(.DestinationCidrBlock | startswith("10.244"))'

# On GCP:
gcloud compute routes list --filter="name~k8s-*"

Service Controller (LoadBalancer Provisioning)

When a Service of type: LoadBalancer is created, the Service Controller calls the cloud's load balancer API to provision an external LB, then writes the allocated IP/hostname back to service.status.loadBalancer.ingress.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    # AWS-specific annotations controlling LB behavior
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:123:certificate/abc"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "ssl"
    # GCP-specific
    cloud.google.com/load-balancer-type: "Internal"
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
  - port: 443
    targetPort: 8443
status:
  loadBalancer:
    ingress:
    - hostname: a1b2c3d4e5.elb.amazonaws.com  # set by CCM after provisioning

Cloud Node Lifecycle Controller

This controller handles the case where a cloud VM is terminated externally (e.g., AWS terminates a spot instance, or an admin deletes a VM in the cloud console). Without this controller, the deleted VM's Node object would remain in the cluster indefinitely, and workloads wouldn't be rescheduled.

# The cloud-node-lifecycle controller polls cloud API for each node
# If the instance no longer exists in the cloud:
# 1. Add taint: node.cloudprovider.kubernetes.io/shutdown:NoSchedule
# 2. After configurable delay, delete the Node object
# This triggers the node lifecycle controller in kcm to evict pods

# Check for nodes in shutdown state
kubectl get nodes -o wide | grep -v Ready
kubectl describe node <node> | grep -A5 "Taints:"

# Spot instance interruption handling (AWS):
# AWS sends 2-minute warning → IMDS endpoint /spot/termination-notice
# Cluster Autoscaler + cloud-node-lifecycle controller coordinate graceful drain

Out-of-Tree Provider Pattern

Every major cloud provider now ships its own CCM binary, versioned and released independently from Kubernetes core. The provider's CCM binary imports k8s.io/cloud-provider and implements the interface.

In-Tree to Out-of-Tree Migration

Clusters created before CCM existed may still use the in-tree cloud providers via --cloud-provider=aws on kube-apiserver and kube-controller-manager. This is deprecated since 1.29 and will be removed. Migration steps:

# Step 1: Update kube-apiserver and kube-controller-manager flags
# Remove: --cloud-provider=aws
# Add:    --cloud-provider=external

# Step 2: Deploy the out-of-tree CCM DaemonSet
# (example: AWS CCM via Helm)
helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
helm upgrade --install aws-cloud-controller-manager \
  aws-cloud-controller-manager/aws-cloud-controller-manager \
  --namespace kube-system \
  --set args={"--v=2","--cloud-provider=aws"}

# Step 3: Verify nodes get cloud metadata from CCM
kubectl get nodes -o yaml | grep -A 10 "addresses:"

# Step 4: CSI migration — replace in-tree volume plugins
# In-tree: kubernetes.io/aws-ebs (deprecated)
# Out-of-tree: ebs.csi.aws.com (CSI driver)
# Migration controlled by feature gate: CSIMigrationAWS=true (default in 1.23+)

# Check if CSI migration is active
kubectl get csidriver ebs.csi.aws.com
kubectl get storageclass | grep ebs

Kubelet and the --cloud-provider=external Flag

Kubelet must be told to wait for CCM to initialize the Node before marking it Ready for scheduling. Without this, pods may be scheduled to a node whose zone/address labels haven't been populated yet.

# kubelet flag required when using CCM:
--cloud-provider=external

# Effect: kubelet adds taint to newly registered node:
# node.cloudprovider.kubernetes.io/uninitialized:NoSchedule
# CCM Node Controller removes this taint after populating cloud metadata
# Only then does the scheduler consider the node for pod placement

# Verify the taint is removed after CCM processes the node
kubectl describe node new-node | grep -A5 Taints:
# Should show: <none> (or only user-defined taints)

# If node stays with uninitialized taint:
kubectl -n kube-system logs <ccm-pod> | grep "node controller"

RBAC, Credentials, and Security

Kubernetes RBAC

CCM needs broad Kubernetes API access to watch and patch Nodes, Services, and Events. A ClusterRole is required:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:cloud-controller-manager
rules:
- apiGroups: [""]
  resources: [events]
  verbs: [create, patch, update]
- apiGroups: [""]
  resources: [nodes]
  verbs: [get, list, watch, delete, patch, update]
- apiGroups: [""]
  resources: [nodes/status]
  verbs: [patch]
- apiGroups: [""]
  resources: [services]
  verbs: [get, list, watch, patch]
- apiGroups: [""]
  resources: [services/status]
  verbs: [update, patch]
- apiGroups: [""]
  resources: [serviceaccounts]
  verbs: [create]
- apiGroups: [""]
  resources: [persistentvolumes]
  verbs: [get, list, update, watch]
- apiGroups: [""]
  resources: [endpoints]
  verbs: [create, get, list, watch, update]
- apiGroups: ["coordination.k8s.io"]
  resources: [leases]
  verbs: [get, create, update]

Cloud Provider Credentials

# Required IAM permissions for AWS CCM:
ec2:DescribeInstances
ec2:DescribeRegions
ec2:DescribeRouteTables
ec2:CreateRoute
ec2:DeleteRoute
ec2:ModifyInstanceAttribute
elasticloadbalancing:* (for Service controller)
autoscaling:DescribeAutoScalingGroups

# GKE Workload Identity annotation
metadata:
  annotations:
    iam.gke.io/gcp-service-account: \
      ccm@my-project.iam.gserviceaccount.com

# Required GCP roles:
roles/compute.instanceAdmin.v1
roles/compute.networkAdmin
roles/iam.serviceAccountUser

# Azure CCM config (cloud-config secret)
apiVersion: v1
kind: Secret
metadata:
  name: cloud-config
  namespace: kube-system
data:
  cloud-config: |
    {
      "cloud": "AzurePublicCloud",
      "useManagedIdentityExtension": true,
      "subscriptionId": "...",
      "resourceGroup": "...",
      "vnetName": "..."
    }

# Flag to pass static credentials
--cloud-config=/etc/kubernetes/cloud-config.json

# Ensure Secrets are encrypted at rest!
# See 01-kube-apiserver.html § Encryption at Rest

Deployment Patterns

DaemonSet on Control Plane Nodes (most common)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cloud-controller-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: cloud-controller-manager
  template:
    metadata:
      labels:
        component: cloud-controller-manager
    spec:
      serviceAccountName: cloud-controller-manager
      tolerations:
      - key: node-role.kubernetes.io/control-plane   # run on control-plane nodes
        effect: NoSchedule
      - key: node.cloudprovider.kubernetes.io/uninitialized  # tolerate uninitialized nodes
        effect: NoSchedule
        value: "true"
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      priorityClassName: system-cluster-critical
      hostNetwork: true          # access cloud IMDS on link-local (169.254.169.254)
      containers:
      - name: cloud-controller-manager
        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.30.0
        command:
        - /bin/aws-cloud-controller-manager
        - --cloud-provider=aws
        - --leader-elect=true
        - --use-service-account-credentials=true
        - --configure-cloud-routes=true  # enable Route Controller
        - --v=2

Configuration Reference

Metrics and Alerting

# Scrape CCM metrics
curl -sk https://localhost:10258/metrics

# Key metrics
cloudprovider_aws_api_request_duration_seconds{request="DescribeInstances"}
cloudprovider_aws_api_request_errors_total{request="CreateLoadBalancer"}

# Controller-generic metrics (same workqueue metrics as kcm)
workqueue_depth{name="cloud-node"}
workqueue_depth{name="service"}
workqueue_depth{name="cloud-node-lifecycle"}
workqueue_retries_total{name="service"}

# Load balancer specific (varies by provider)
cloudprovider_aws_api_throttled_requests_total   # AWS throttling events
cloud_provider_reconcile_attempts_total{provider="aws",controller="service"}

# Node initialization health
# Monitor: nodes stuck with uninitialized taint > 2 minutes
kubectl get nodes -o json | jq -r '
  .items[] |
  select(.spec.taints != null) |
  select(.spec.taints[] | .key == "node.cloudprovider.kubernetes.io/uninitialized") |
  .metadata.name
'

groups:
- name: cloud-controller-manager
  rules:
  - alert: CCMDown
    expr: absent(up{job="cloud-controller-manager"} == 1)
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "cloud-controller-manager is down"
      description: "No CCM is running. LoadBalancer Services will not be provisioned and new nodes will stay uninitialized."

  - alert: CCMNodeUninitializedStuck
    expr: |
      kube_node_spec_taint{key="node.cloudprovider.kubernetes.io/uninitialized"} > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Node stuck with cloud-uninitialized taint"
      description: "Node {{ $labels.node }} has been uninitialized for >5m. Check CCM logs."

  - alert: CCMCloudAPIErrors
    expr: rate(cloudprovider_aws_api_request_errors_total[5m]) > 0.1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High CCM cloud API error rate"
      description: "CCM is seeing {{ $value }} cloud API errors/s on {{ $labels.request }}."

  - alert: CCMLoadBalancerSyncFailed
    expr: rate(workqueue_retries_total{job="cloud-controller-manager",name="service"}[5m]) > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Service LB sync failing repeatedly"
      description: "CCM service controller is retrying LB sync at {{ $value }}/s."

Troubleshooting

# Check CCM is running
kubectl -n kube-system get pods -l component=cloud-controller-manager

# Check CCM logs for the service
kubectl -n kube-system logs -l component=cloud-controller-manager | grep -i "service\|loadbalancer\|error" | tail -30

# Check service events
kubectl describe service my-svc | grep -A 10 Events:
# Look for: "Error creating load balancer", "Timeout", "Throttling"

# AWS: check IAM permissions
# "AccessDenied" errors indicate missing IAM policy on control plane node role
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::ACCOUNT:role/k8s-control-plane \
  --action-names "elasticloadbalancing:CreateLoadBalancer" \
  --resource-arns "*"

# GCP: check service account permissions
gcloud projects get-iam-policy my-project \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:ccm@my-project.iam.gserviceaccount.com"

# Check CCM logs for node initialization
kubectl -n kube-system logs <ccm-pod> | grep "node controller\|initialize\|providerID"

# Verify providerID is set on the node (required for CCM lookup)
kubectl get node <node-name> -o jsonpath='{.spec.providerID}'
# Should be: aws:///us-east-1a/i-0abc123def456
# If empty, kubelet is missing --provider-id or --cloud-provider=external

# Check IMDS access from CCM pod (for IAM role fetching)
kubectl -n kube-system exec <ccm-pod> -- \
  curl -s http://169.254.169.254/latest/meta-data/instance-id
# Must return an instance ID; failure = network/IMDS issue

# Check if Node has valid providerID format for the cloud
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name): \(.spec.providerID)"'

# Verify Route Controller is enabled
kubectl -n kube-system logs <ccm-pod> | grep -i "route"

# Check nodes have podCIDR assigned
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name): \(.spec.podCIDR)"'

# AWS: verify routes in VPC route table
aws ec2 describe-route-tables \
  --filter Name=vpc-id,Values=vpc-0abc123 \
  --query 'RouteTables[].Routes[?InstanceId!=null]'

# If using Calico VXLAN or Flannel VXLAN, routes are NOT needed:
# Set --configure-cloud-routes=false to disable Route Controller

# Check if cloud CIDR conflicts with VPC CIDR
# Pod CIDR should not overlap with VPC subnet CIDRs

# AWS: EC2 API has per-region throttling limits
# Reduce polling frequency:
# --node-sync-period=30s (default 10s)
# --route-reconciliation-period=30s

# Check for throttling in CCM logs
kubectl -n kube-system logs <ccm-pod> | grep -i "throttl\|rateLim\|RequestLimitExceeded"

# AWS: request a limit increase for EC2 API calls via AWS console

# GCP: Cloud Resource Manager API quotas
# Enable API rate limiting in cloud config:
# rateLimitConfig:
#   cloudProviderRateLimit: true
#   cloudProviderRateLimitQPS: 3
#   cloudProviderRateLimitBucket: 5

# Monitor cloud API call rate
kubectl get --raw /metrics | grep cloudprovider_.*_api_request_duration

On-Premises Clusters Without CCM

Clusters running on bare metal, VMware, or private cloud without a CCM must handle the functionality themselves or accept limitations.

Production Best Practices