Garbage Collection Flow
Overview
Traces what happens when an owner object is deleted — how ownerReferences drive cascade deletion, how the garbage collector controller resolves dependency graphs, and how finalizers block or gate cleanup.
Garbage Collection Architecture
Kubernetes GC is graph-based: every object can declare owners via ownerReferences.
When an owner is deleted, dependents are collected — either immediately (foreground)
or in the background.
Owner Dependent
┌─────────┐ ┌───────────────────┐
│Deployment│──owns───►│ReplicaSet │──owns───►│Pod│
└─────────┘ │ownerRef: │ └───┘
│ name: payments-api│
│ uid: abc-123 │
└───────────────────┘
GC Controller (in kube-controller-manager):
- Maintains an in-memory directed acyclic graph of all objects + ownerRefs
- Watches ALL object types via informers
- When owner deleted → enqueues dependents for deletionGarbage Collection Sequence — Background Deletion (Default)
kubectl API Server etcd GC Controller
│ │ │ │
│─DELETE deploy────►│ │ │
│ (no propagation │ │ │
│ policy set) │ │ │
│ │──DELETE ─────►│ │
│◄── 200 OK ────────│ │ │
│ │ │ │
│ │──WATCH event ──────────────► │
│ │ (Deployment Deleted) │
│ │ │
│ │ ┌─── GC controller ────────────┐
│ │ │ Look up dependents of │
│ │ │ Deployment uid=abc-123: │
│ │ │ RS: payments-api-v2 (bound)│
│ │ │ RS: payments-api-v1 (old) │
│ │ └──────────────────────────────┘
│ │ │
│ │◄── DELETE RS payments-api-v2 ─│
│ │◄── DELETE RS payments-api-v1 ─│
│ │──DELETE RSs ──────►│ │
│ │ │ │
│ │──WATCH event ──────────────► │
│ │ (ReplicaSet Deleted) │
│ │ │
│ │ ┌─── GC: RS dependents ───────┐
│ │ │ Pods owned by RS: │
│ │ │ pod-xxx, pod-yyy, pod-zzz │
│ │ └──────────────────────────────┘
│ │◄── DELETE pod-xxx, yyy, zzz ──│
│ │──DELETE pods ─────►│ │
│ │ │ │
│ Deployment → RSs → Pods all gone
│ (async, order not guaranteed)Foreground Deletion Sequence
kubectl API Server etcd GC Controller
│ │ │ │
│─DELETE deploy────►│ │ │
│ propagationPolicy│ │ │
│ : Foreground │ │ │
│ │ │ │
│ ┌─── API Server sets ──────────────┐ │
│ │ metadata.deletionTimestamp: now │ │
│ │ metadata.finalizers: │ │
│ │ [foregroundDeletion] │ │
│ │ (object NOT deleted from etcd │ │
│ │ until finalizer removed) │ │
│ └───────────────────────────────────┘ │
│◄── 200 OK (object is "terminating") ─────────────│
│ │ │ │
│ │──WATCH ────────────────────► │
│ │ (Deployment Modified — │
│ │ deletionTimestamp set) │
│ │ │
│ │ ┌─── GC: foreground ──────────┐
│ │ │ Find all "blocking" owners │
│ │ │ (ownerRef.blockOwnerDeletion=true)
│ │ │ → RS payments-api-v2 │
│ │ │ Delete RSs first │
│ │ │ RS deletes its Pods │
│ │ │ Once all dependents gone: │
│ │ │ PATCH deploy.finalizers = [] │
│ │ └──────────────────────────────┘
│ │◄── PATCH finalizers=[] ───────│
│ │──DELETE deploy ───►│ │
│ │ │ │
│ Deployment deleted only AFTER all dependents gone
│ (kubectl delete blocks until owner is gone)Orphan Deletion
kubectl delete deployment payments-api \
--cascade=orphan (or propagationPolicy: Orphan)
Effect:
- Deployment deleted immediately
- ReplicaSets and Pods have their ownerReference REMOVED
- They become "orphaned" — no owner — and continue running
- GC controller does NOT delete orphaned objects
Use cases:
- Adopt existing pods with a new Deployment
- Delete a StatefulSet without destroying its PVCs
- Debug: keep pods running after deleting the controllerownerReferences in Practice
# ReplicaSet automatically created with ownerReference to Deployment
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: payments-api-abc123
ownerReferences:
- apiVersion: apps/v1
kind: Deployment
name: payments-api
uid: abc-123-def-456 # UID, not just name — immutable across recreates
controller: true # this owner controls the object
blockOwnerDeletion: true # this RS blocks foreground deletion of DeploymentownerReference rules:
- uid must match the owner's current UID
(prevents stale references after recreate with same name)
- Cross-namespace ownerReferences are not allowed
(Kubernetes GC rejects them — namespace-scoped objects cannot own cluster-scoped)
- Multiple owners allowed (but only one controller:true per object)Finalizers — Blocking Deletion
Finalizers are strings in metadata.finalizers[].
An object with finalizers set will NOT be deleted from etcd
until all finalizers are removed, regardless of deletionTimestamp.
Flow:
kubectl delete secret payments-db-creds
→ API server: sets deletionTimestamp, leaves finalizers intact
→ External controller watching for deletionTimestamp:
performs cleanup (rotate secret in Vault, notify audit system)
PATCH secret: metadata.finalizers = []
→ API server sees empty finalizers + deletionTimestamp set
→ Deletes object from etcd# External secrets controller pattern: finalizer on Secret
apiVersion: v1
kind: Secret
metadata:
name: payments-db-creds
namespace: production
finalizers:
- secrets.external-secrets.io/cleanup # registered by ESO controller
# When this secret is deleted:
# 1. ESO controller sees deletionTimestamp
# 2. Removes the external secret from Vault
# 3. Removes this finalizer
# 4. K8s deletes the Secret objectGC Controller Internals
Graph builder (goroutine):
Watches ALL object types
On Add: insert node, link to owner nodes
On Update: update edges (ownerRefs changed)
On Delete: mark node as "virtual deleted" (may have dependents)
GC processor (goroutine):
Processes "dirty queue" of nodes to potentially delete
For each dirty node:
if node has no owner AND no parent in graph → skip (root object)
if all owners are deleted → delete this node
if owner exists but ownerRef.uid mismatch → ownerRef is stale → treat as orphan
Resync: full graph rebuild every 30 seconds to catch missed watch eventsCascading Delete Comparison
| Mode | Flag / Policy | Owner deleted when | Dependents deleted when |
|---|---|---|---|
| Background (default) | --cascade=background | Immediately | Asynchronously by GC |
| Foreground | --cascade=foreground | After all blocking dependents gone | Synchronously before owner |
| Orphan | --cascade=orphan | Immediately | Never (ownerRef removed) |
Debugging GC Issues
# Check if object is stuck terminating (has finalizers)
kubectl get pod payments-api-xxx -n production -o jsonpath='{.metadata.finalizers}'
# Check deletionTimestamp
kubectl get deploy payments-api -n production \
-o jsonpath='{.metadata.deletionTimestamp}'
# Force-remove a finalizer (last resort — bypasses controller cleanup)
kubectl patch pod payments-api-xxx -n production \
-p '{"metadata":{"finalizers":[]}}' --type=merge
# Check owner references on a ReplicaSet
kubectl get rs -n production -o json | \
jq '.items[].metadata | {name:.name, owners:.ownerReferences}'
# List orphaned ReplicaSets (no owner Deployment)
kubectl get rs -n production -o json | \
jq '.items[] | select(.metadata.ownerReferences == null) | .metadata.name'
# GC controller logs
kubectl logs -n kube-system -l component=kube-controller-manager \
--tail=100 | grep -i "garbage\|gc\|orphan"
# Check if GC is keeping up
kubectl get --raw /metrics | grep garbagecollectorCommon GC Problems
Problem: Object stuck terminating
Cause: finalizer registered but controller that owns it is gone
Fix: identify which controller handles the finalizer, restore it
or manually remove finalizer as last resort
Problem: Orphaned Pods after Deployment delete
Cause: used --cascade=orphan intentionally or a bug in controller
Fix: kubectl delete pods -l app=payments-api -n production
Problem: PVC not deleted after StatefulSet delete
Cause: StatefulSets intentionally do NOT own their PVCs
(prevents accidental data loss on StatefulSet delete)
Fix: PVCs must be deleted manually after StatefulSet removal
Problem: ReplicaSet not garbage collected
Cause: ownerReference.uid mismatch (Deployment was recreated with same name)
Fix: old RS still has uid of deleted Deployment; GC cannot find owner
→ manually delete stale RSRelated
- 01 — Pod Creation Flow — ownerRefs set during pod creation
- 05 — Rolling Update Flow — old ReplicaSets kept for rollback, then GC'd
- 07 — CSI Flow — PV protection finalizer prevents PV deletion while bound