CSI Flow
Overview
Traces the full journey from kubectl apply of a PVC to a pod having a formatted, mounted filesystem — through every CSI component and Kubernetes controller that participates.
CSI Component Map
kube-controller-manager
├── PersistentVolume controller — binds PVCs to PVs
├── AttachDetach controller — manages VolumeAttachment objects
└── PV protection controller — finalizers on bound PVCs/PVs
CSI External Components (run as sidecars alongside CSI controller pod):
├── external-provisioner — watches PVCs, calls CreateVolume
├── external-attacher — watches VolumeAttachments, calls ControllerPublishVolume
├── external-resizer — watches PVCs, calls ControllerExpandVolume
└── external-snapshotter — watches VolumeSnapshots, calls CreateSnapshot
CSI Driver — Controller Plugin (Deployment, 1+ replicas):
└── CSI gRPC server
├── CreateVolume — calls AWS EBS CreateVolume API
├── DeleteVolume — calls AWS EBS DeleteVolume API
├── ControllerPublishVolume — calls AWS EBS AttachVolume API
└── ControllerUnpublishVolume — calls AWS EBS DetachVolume API
CSI Driver — Node Plugin (DaemonSet, 1 per node):
└── CSI gRPC server
├── NodeStageVolume — format filesystem + mount to staging path
├── NodePublishVolume — bind-mount staged path into pod
├── NodeUnpublishVolume — unmount from pod
└── NodeExpandVolume — resize filesystem after controller expansionFull CSI Sequence — PVC Creation to Pod Mount
kubectl API Server etcd PV Controller external-provisioner CSI Controller CSI Node kubelet
│ │ │ │ │ │ │ │
│─POST PVC──────►│ │ │ │ │ │ │
│ (storageClass: │ │ │ │ │ │ │
│ gp3-retain) │ │ │ │ │ │ │
│ │──WRITE───────►│ │ │ │ │ │
│◄─201 Created───│ │ │ │ │ │ │
│ PVC phase: │ │ │ │ │ │ │
│ Pending │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │──WATCH PVC ──────────────►│ │ │ │ │
│ │ (PVC unbound) │ │ │ │ │
│ │ │ [WaitForFirstConsumer: │ │ │
│ │ │ wait for pod to be scheduled] │ │ │
│ │ │ │ │ │ │
│─POST Pod──────►│ │ │ │ │ │
│ (uses PVC) │──WRITE───────►│ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ [scheduler selects node in AZ-b] │ │ │ │
│ │ [updates pod.spec.nodeName] │ │ │ │
│ │ │ │ │ │ │
│ │──WATCH PVC ──────────────────────────────►│ │ │ │
│ │ (unbound PVC + pod scheduled in AZ-b) │ │ │ │
│ │ │ │ │ │
│ │ │─CreateVolume──────►│ │ │
│ │ │ (AZ: us-east-1b, │ │ │
│ │ │ size: 100Gi, │ │ │
│ │ │ type: gp3) │ │ │
│ │ │ │── AWS API ──► │ │
│ │ │ │ EC2 │ │
│ │ │ │ CreateVolume│ │
│ │ │◄── vol-0abc123 ────│ │ │
│ │ │ │ │ │
│ │◄── CREATE PV ─────────────────────────────│ │ │ │
│ │ (provisioned PV for vol-0abc123) │ │ │ │
│ │──WRITE PV ────►│ │ │ │ │
│ │ │ │ │ │ │
│ │──WATCH PV ───────────────►│ │ │ │ │
│ │ (PV Available) │ │ │ │ │
│ │ │ [PV controller binds PVC→PV] │ │ │
│ │◄── PATCH PVC phase:Bound ─│ │ │ │ │
│ │◄── PATCH PV claimRef ─────│ │ │ │ │
│ │──WRITE PVC+PV ────────────►(etcd) │ │ │ │
│ │ │ │ │ │
│ │ [AttachDetach controller sees pod+PV] │ │ │ │
│ │◄── CREATE VolumeAttachment ───────────────│ │ │ │
│ │──WRITE VA ────►│ │ │ │ │
│ │ │ │ │ │
│ │──WATCH VA ──────────────────────────────────────────────────► │ │ │
│ │ (VolumeAttachment created) │ │ │ │
│ │ │─ControllerPublish─►│ │ │
│ │ │ Volume(vol-0abc, │ │ │
│ │ │ node=worker-3) │── AWS API ──►│ │
│ │ │ │ AttachVolume│ │
│ │ │◄── attached ───────│ │ │
│ │◄── PATCH VA.status.attached=true ─────────│ │ │ │
│ │──WRITE VA ────►│ │ │ │ │
│ │ │ │ │
│ │──WATCH VA.attached ───────────────────────────────────────────────────────────────────► │
│ │ │ │
│ │ [kubelet sees pod + PVC] │
│ │ │─NodeStage►│
│ │ │ Volume │
│ │ │ (format ext4,
│ │ │ mount /var/lib/kubelet/plugins/...)
│ │ │◄─staged───│
│ │ │ │
│ │ │─NodePublish►
│ │ │ Volume │
│ │ │ (bind-mount
│ │ │ into pod /data)
│ │ │◄─published│
│ │ │ │
│ │ [kubelet starts containers] │
pod running with /data mounted from EBS vol-0abc123 │Key State Transitions
| Object | State before | Event | State after |
|---|---|---|---|
| PVC | Pending | PV provisioned | Bound |
| PV | Available | PVC bound | Bound |
| VolumeAttachment | (created) | ControllerPublishVolume | .status.attached: true |
| PVC | Bound | Pod terminated, PV reclaim=Retain | Bound (PV Released) |
| PV | Bound | PVC deleted | Released (Retain) or deleted (Delete) |
CSI Staging Path
Node filesystem layout (CSI convention):
/var/lib/kubelet/
plugins/
kubernetes.io/csi/
ebs.csi.aws.com/
volumeDevices/
publish/
<pv-name>/ ← block device (raw)
plugins_registry/ ← CSI node plugin socket registration
pods/
<pod-uid>/
volumes/
kubernetes.io~csi/
<pv-name>/
mount/ ← bind-mount target (NodePublishVolume)
The staging path allows multiple pods to share the same volume
(for RWX) by staging once and publishing (bind-mounting) multiple times.CSI Node Plugin Socket Registration
kubelet plugin watcher watches: /var/lib/kubelet/plugins_registry/
CSI node plugin:
1. Creates socket: /var/lib/kubelet/plugins/<driver-name>/csi.sock
2. Calls Registration service on kubelet:
RegisterPlugin(name="ebs.csi.aws.com", endpoint="csi.sock")
3. kubelet validates and stores the socket path
kubelet uses this socket for all NodeStageVolume / NodePublishVolume callsDebugging CSI Issues
# VolumeAttachment not becoming attached
kubectl get volumeattachment -o wide
kubectl describe volumeattachment <name>
# Look for: "Error attaching volume" or "Multi-Attach error"
# Check external-attacher logs
kubectl logs -n kube-system \
-l app=ebs-csi-controller -c csi-attacher --tail=50
# Check AWS API calls from CSI controller
kubectl logs -n kube-system \
-l app=ebs-csi-controller -c ebs-plugin --tail=100
# NodeStageVolume failing (pod stuck ContainerCreating)
TARGET_NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
kubectl logs -n kube-system \
$(kubectl get pod -n kube-system -l app=ebs-csi-node \
--field-selector spec.nodeName=$TARGET_NODE \
-o jsonpath='{.items[0].metadata.name}') \
-c ebs-plugin --tail=100
# Check volume is attached from AWS side
PV_HANDLE=$(kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeHandle}')
aws ec2 describe-volumes --volume-ids $PV_HANDLE \
--query 'Volumes[0].{State:State,Attachments:Attachments}'Related
- 06 — Storage Operations — storage ops playbook
- 04 — CSI Drivers — CSI driver reference
- 14 — Volume Attach Flow — detailed attach/detach controller flow