CNI Setup Flow
Overview
Traces the complete network setup sequence for a new pod — from kubelet creating a network namespace through CNI plugin invocation, IPAM address allocation, veth pair creation, and the pod having a routable IP.
CNI Architecture
kubelet is NOT responsible for networking.
CNI plugins are responsible — kubelet calls them via the CNI spec interface.
Pod network setup:
1. kubelet creates network namespace (pause container)
2. kubelet calls CNI ADD with netns path + config
3. CNI plugin configures: IP, routes, veth pair, iptables/BPF
4. kubelet starts app containers in the configured namespace
CNI plugin location:
/opt/cni/bin/<plugin> — executables
/etc/cni/net.d/<config>.conf — JSON configuration
CNI call interface:
Environment variables:
CNI_COMMAND = ADD | DEL | CHECK | VERSION
CNI_CONTAINERID = pause container ID
CNI_NETNS = /proc/<pid>/ns/net (or /var/run/netns/...)
CNI_IFNAME = eth0
CNI_PATH = /opt/cni/bin
stdin: JSON config (from /etc/cni/net.d/)Full CNI Setup Sequence — Cilium
kubelet containerd CNI Plugin (Cilium) Cilium Agent Linux Kernel
│ │ │ │ │
│ [Pod scheduled to this node] │ │ │
│ │ │ │ │
│─RunPodSandbox──►│ │ │ │
│ (create pause │ │ │ │
│ container) │ │ │ │
│ │ │ │ │
│ │──clone(NEWNET) ──────────────────────────────────────►│
│ │ (create network namespace) │ │
│ │◄── netns: /proc/12345/ns/net ─────────────────────────│
│ │ │ │ │
│◄── SandboxID ───│ │ │ │
│ │ │ │ │
│ [kubelet calls CNI ADD] │ │ │
│──CNI_ADD ───────────────────────► │ │ │
│ CNI_COMMAND=ADD │ │ │
│ CNI_CONTAINERID=abc123 │ │ │
│ CNI_NETNS=/proc/12345/ns/net │ │ │
│ CNI_IFNAME=eth0 │ │ │
│ stdin: cilium CNI config │ │ │
│ │ │ │ │
│ │ ┌─── Cilium CNI plugin ───────────────────┐ │
│ │ │ 1. Contact cilium-agent via UNIX socket │ │
│ │ │ /var/run/cilium/cilium.sock │ │
│ │ └──────────────────────────────────────────┘ │
│ │ │──CmdAdd RPC ──────►│ │
│ │ │ (containerID, │ │
│ │ │ netns, ifName) │ │
│ │ │ │ │
│ │ │ ┌─── Cilium Agent ──────┐│
│ │ │ │ Allocate IP from IPAM││
│ │ │ │ (via CiliumNode CRD ││
│ │ │ │ or AWS ENI pool) ││
│ │ │ │ → 10.0.1.47/32 ││
│ │ │ └───────────────────────┘│
│ │ │ │──ip addr add──►│
│ │ │ │ 10.0.1.47 │
│ │ │ │ in netns │
│ │ │ │ │
│ │ │ │──ip link add──►│
│ │ │ │ veth pair: │
│ │ │ │ lxcXXX (host) │
│ │ │ │ eth0 (pod ns) │
│ │ │ │ │
│ │ │ │──BPF programs─►│
│ │ │ │ attached to │
│ │ │ │ lxcXXX │
│ │ │ │ (tc ingress + │
│ │ │ │ egress) │
│ │ │ │ │
│ │ │◄── CmdAdd OK ──────│ │
│ │ │ {IP: 10.0.1.47, │ │
│ │ │ GW: 10.0.1.1, │ │
│ │ │ routes: [...]} │ │
│◄── CNI result ──────────────────── │ │ │
│ {ips:[{address:"10.0.1.47/32"}]} │ │ │
│ │ │ │ │
│ [kubelet writes pod.status.podIP = 10.0.1.47] │ │
│ [kubelet starts app containers in the namespace] │ │Network Namespace — What Gets Created
Inside pod network namespace (after CNI ADD):
Interfaces:
eth0 — veth end, IP 10.0.1.47/32
lo — loopback 127.0.0.1
Routes:
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link (Cilium gateway IP, ARP proxied)
On host (node):
lxcXXXXXX — veth peer (host end)
BPF programs attached:
tc filter ingress → cilium_to_container
tc filter egress → cilium_from_container
Traffic flow (pod → pod on different node):
pod eth0 → kernel routes to 169.254.1.1
→ BPF at lxcXXX: lookup destination in tunnel/route map
→ VXLAN/Geneve encapsulation OR direct routing (depends on mode)
→ node NIC → wire → destination node NIC → BPF decap → destination podIPAM Modes
Cluster-scope IPAM (default Cilium):
- Cilium agent manages a pool of pod CIDRs per node
- Assigned via CiliumNode CRD: .spec.ipam.podCIDRs
- No cloud API calls per pod IP
- Fast allocation (in-memory pool)
AWS ENI IPAM (Cilium ENI mode / aws-vpc-cni):
- Each pod gets a real VPC IP from an ENI secondary IP
- Slower: requires EC2 API calls to warm IP pool
- Benefit: pods are routable in VPC without overlay
AWS VPC CNI (amazon-vpc-cni / aws-node DaemonSet):
- IPAMD daemon manages ENI IP pools per node
- L-IPAM: warm pool pre-allocates IPs in background
- Pod gets VPC IP → routable natively in AWS VPC, to RDS, etc.CNI Configuration Files
// /etc/cni/net.d/05-cilium.conf (Cilium)
{
"cniVersion": "0.3.1",
"name": "cilium",
"type": "cilium-cni",
"enable-debug": false
}
// Cilium CNI binary delegates everything to cilium-agent
// via UNIX socket — config is minimal
// /etc/cni/net.d/10-aws.conf (aws-vpc-cni)
{
"cniVersion": "0.3.1",
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni",
"mtu": 9001,
"pluginLogLevel": "DEBUG",
"pluginLogFile": "/var/log/aws-routed-eni/plugin.log"
}
// /etc/cni/net.d/10-flannel.conf (Flannel — no NetworkPolicy)
{
"cniVersion": "0.3.1",
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true}
}
]
}Pod Deletion — CNI DEL Sequence
kubelet CNI Plugin Cilium Agent Linux Kernel
│ │ │ │
│ [Pod deleted] │ │ │
│──CNI_DEL ──────►│ │ │
│ CNI_COMMAND=DEL│ │ │
│ CNI_CONTAINERID=abc123 │ │
│ CNI_NETNS=/proc/12345/ns/net │ │
│ │──CmdDel RPC ─────►│ │
│ │ │ │
│ │ [Cilium agent:] │
│ │ Remove BPF endpoint maps │
│ │ Release IP 10.0.1.47 back to pool
│ │ Delete veth pair lxcXXX ─────────►│
│ │ Remove from endpoint list │ │
│ │◄── DEL OK ────────│ │ │
│◄── CNI DEL OK ──│ │ │ │
│ │ │ │ │
│ [kubelet deletes network namespace]│ │──del ns─►│Multi-Network Pods (Multus)
Multus is a "meta" CNI plugin that calls multiple CNI plugins:
/etc/cni/net.d/00-multus.conf
→ Multus reads Pod annotation: k8s.v1.cni.cncf.io/networks
→ Calls primary CNI (Cilium) → eth0 with cluster IP
→ Calls secondary CNI (SRIOV) → net1 with DPDK interface
→ Returns combined result
Use cases:
- Storage traffic on separate interface (SRIOV)
- SR-IOV high-throughput for NFV workloads
- Telco 5G pods needing dedicated NICDebugging CNI Issues
# Pod stuck in ContainerCreating — check CNI
kubectl describe pod <pod> -n <ns>
# Events: Failed to create pod sandbox: rpc error: code = Unknown
# failed to setup network for sandbox ... CNI plugin returned error
# Check CNI plugin logs (Cilium)
kubectl logs -n kube-system -l k8s-app=cilium --tail=50 | grep -i "cni\|ipam\|error"
# Check aws-vpc-cni IPAMD logs
kubectl logs -n kube-system -l k8s-app=aws-node -c aws-node --tail=50
# Verify CNI binary exists on node
kubectl debug node/<node-name> -it --image=ubuntu -- \
ls /opt/cni/bin/
# Check CNI config
kubectl debug node/<node-name> -it --image=ubuntu -- \
cat /etc/cni/net.d/05-cilium.conf
# Verify pod IP assigned
kubectl get pod <pod> -o jsonpath='{.status.podIP}'
# Cilium: check endpoint status for pod
CILIUM_POD=$(kubectl get pod -n kube-system -l k8s-app=cilium \
--field-selector spec.nodeName=<node> \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system $CILIUM_POD -- cilium endpoint list
# Check IP allocation pool usage
kubectl exec -n kube-system $CILIUM_POD -- cilium ip list
# AWS VPC CNI: check ENI warm pool
kubectl exec -n kube-system <aws-node-pod> -- \
/app/grpc-health-probe -addr=localhost:50051
kubectl logs -n kube-system <aws-node-pod> -c aws-node | grep "warm pool"Related
- 08 — Network Policy Flow — BPF programs loaded by CNI for policy enforcement
- 02 — CNI Plugins — Cilium/Calico/Flannel comparison
- 02 — Network Issues — CNI troubleshooting playbook