Kubernetes API Model

REST API URL Patterns

Every Kubernetes resource is addressable by a deterministic URL. kube-apiserver (see 01-kube-apiserver.html) serves these paths over HTTPS on :6443.

Subresources

# Inspect raw API calls made by kubectl
kubectl get pod nginx -n default -v=8 2>&1 | grep -E 'GET|POST|PATCH'

# Equivalent curl (service-account token from inside a pod)
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -sk https://kubernetes.default.svc/api/v1/namespaces/default/pods/nginx \
  -H "Authorization: Bearer $TOKEN"

# Discover all API groups + versions
kubectl api-versions | sort
curl -sk https://<apiserver>:6443/apis --cert client.crt --key client.key \
  --cacert ca.crt | jq '.groups[].name'

API Groups and Versions

Resources are organized into named API groups with independent versioning lifecycles. The core group (/api/v1) is legacy; all modern resources live under named groups in /apis/.

Resource Object Anatomy

Every Kubernetes resource is a typed JSON/YAML object with a universal envelope. Understanding each field is essential for writing operators and debugging API behavior.

apiVersion: apps/v1          # <group>/<version> — TypeMeta
kind: Deployment             # PascalCase resource kind — TypeMeta
metadata:
  name: nginx                # required unless generateName is set
  generateName: nginx-       # server appends 5-char random suffix; used by ReplicaSets
  namespace: default
  uid: 3fa85f64-5717-4562-b3fc-2c963f66afa6   # immutable UUID set by server
  resourceVersion: "1042819" # etcd revision string — optimistic locking & watch
  generation: 3              # incremented on spec changes (NOT status updates)
  creationTimestamp: "2024-01-15T10:30:00Z"
  deletionTimestamp: null    # set when DELETE is called; lives until finalizers clear
  deletionGracePeriodSeconds: 30
  labels:
    app: nginx
    version: "1.25"
  annotations:
    deployment.kubernetes.io/revision: "3"
    kubectl.kubernetes.io/last-applied-configuration: '...'
  ownerReferences:           # garbage-collection chain
  - apiVersion: apps/v1
    kind: ReplicaSet
    name: nginx-6799fc88d8
    uid: abc123
    controller: true         # only one ownerRef may have controller:true
    blockOwnerDeletion: true # prevents owner deletion until dependent gone
  finalizers:                # object not deleted until all strings removed
  - kubernetes.io/pvc-protection
  managedFields:             # Server-Side Apply field ownership tracking
  - manager: kubectl
    operation: Apply
    apiVersion: apps/v1
    time: "2024-01-15T10:30:00Z"
    fieldsType: FieldsV1
    fieldsV1: { ... }
spec:                        # desired state — varies by kind
  replicas: 3
status:                      # observed state — written only by controllers
  readyReplicas: 3

generateName and Naming Conventions

metadata.generateName causes kube-apiserver to append a random 5-character alphanumeric suffix before persisting the object. Used by ReplicaSets (which create Pods with generateName: nginx-6799fc88d8-), Jobs, and any controller that needs guaranteed unique names.

• If both name and generateName are set, name wins — generateName is ignored.
• All resource names must conform to DNS subdomain rules (RFC 1123): lowercase alphanumeric or -, max 253 chars, start/end alphanumeric — except Pods which use DNS label rules (max 63 chars per segment).
• The server rejects names violating these rules with 422 Unprocessable Entity.
• Namespace names are DNS labels (max 63 chars). Avoid dots in namespace names — they cause DNS resolution ambiguity.

# generateName in action
kubectl run --generate-name=debug- --image=busybox --restart=Never -- sleep 3600
# Result: debug-k7x9p (or similar)

# DNS subdomain validation — this will fail
kubectl create namespace "My_Namespace"
# Error: namespace name must match DNS-1123 subdomain pattern

HTTP Verbs, RBAC Verbs, and Patch Types

Four Patch Types

kubectl patch pod nginx \
  --type=merge \
  -p '{"spec":{"terminationGracePeriodSeconds":60}}'

kubectl patch deployment nginx \
  --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx:1.26"}]}}}}'

kubectl patch deployment nginx \
  --type=json \
  -p '[
    {"op":"test","path":"/spec/replicas","value":3},
    {"op":"replace","path":"/spec/replicas","value":5}
  ]'

kubectl apply --server-side \
  --field-manager=my-operator \
  -f deployment.yaml

DeleteOptions and Propagation Policy

When deleting an object, the client can send a DeleteOptions body to control cascading behavior and race protection:

apiVersion: v1
kind: DeleteOptions
gracePeriodSeconds: 0         # override pod's terminationGracePeriodSeconds (0 = immediate)
preconditions:
  uid: "3fa85f64-..."         # only delete if UID matches — prevents deleting replacement
  resourceVersion: "1042819"  # only delete if resourceVersion matches — prevents races
propagationPolicy: Foreground # Foreground | Background | Orphan

Watch Protocol and ResourceVersion

Controllers never poll. They establish a long-lived HTTP/2 GET with ?watch=true that streams NDJSON event objects. The resourceVersion field drives resumable watches and optimistic locking.

ResourceVersion and MVCC

Every object and collection has a resourceVersion — a string representation of the etcd revision at which it was last written. The MVCC internals are covered in 02-etcd.html § MVCC. From the API perspective:

• Optimistic locking: PUT/PATCH must include the current resourceVersion; server returns 409 Conflict if another writer changed it first.
• Watch resumption: ?resourceVersion=X streams all events after revision X. If X is compacted, returns 410 Gone.
• Cache reads: ?resourceVersion=0 reads from watchCache (slightly stale but fast). Omit for quorum read from etcd.

Informer Architecture

Controllers never call the API per event. The client-go informer machinery consolidates List+Watch into a local in-memory store and work queue, enabling safe concurrent reconciliation. The server-side counterpart — watchCache — is covered in 01-kube-apiserver.html § watchCache.

import (
    ctrl "sigs.k8s.io/controller-runtime"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
)

type MyReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Read from local cache — NOT a live API call
    var myObj myv1.MyResource
    if err := r.Get(ctx, req.NamespacedName, &myObj); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil // deleted; nothing to do
        }
        return ctrl.Result{}, err
    }

    if myObj.Status.Phase == "" {
        myObj.Status.Phase = "Pending"
        // Status update only touches the /status subresource
        if err := r.Status().Update(ctx, &myObj); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Periodic re-check
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

// Register with controller-manager
ctrl.NewControllerManagedBy(mgr).
    For(&myv1.MyResource{}).
    Owns(&appsv1.Deployment{}).  // watch Deployments owned by MyResource
    Complete(r)

Server-Side Apply (SSA)

SSA moves merge logic to the server and introduces field ownership — each field is owned by a named manager. This enables safe co-management by multiple actors (controller + HPA + human operator).

# Apply with SSA
kubectl apply --server-side -f deployment.yaml

# Custom manager name for operators
kubectl apply --server-side --field-manager=my-operator -f crd-instance.yaml

# Force-take ownership of conflicting fields
kubectl apply --server-side --force-conflicts -f deployment.yaml

# Inspect managed fields
kubectl get deployment nginx -o jsonpath='{.metadata.managedFields}' | jq .

# Remove a field from SSA management: omit it from the manifest and re-apply
# The field becomes "unmanaged" and other managers may adopt it

etcd Key Encoding

All objects are persisted in etcd under deterministic key paths. Deep internals (MVCC, compaction, WAL) are in 02-etcd.html § Kubernetes Keyspace.

# Key examples
/registry/pods/default/nginx-abc12
/registry/deployments/kube-system/coredns
/registry/secrets/default/my-secret
/registry/clusterroles/admin                                    # cluster-scoped: no namespace
/registry/apiextensions.k8s.io/customresourcedefinitions/foos.example.com

# Object encoding: k8s\x00 magic prefix + protobuf Unknown wrapper
# To decode (needs etcd access + auger tool):
etcdctl get /registry/pods/default/nginx --print-value-only | auger decode

# kube-apiserver uses a different prefix for each API group:
# /registry/{group}/{resource}/{ns}/{name}
# Core group omits the group: /registry/pods/...

API Request Lifecycle

Every request traverses a staged pipeline in kube-apiserver. Full detail in 01-kube-apiserver.html § Request Pipeline.

Label and Field Selectors

Label Selectors

kubectl get pods -l app=nginx,tier=frontend
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l '!canary'  # pods WITHOUT the canary label

# ReplicaSet/Deployment combined selector
selector:
  matchLabels:
    app: nginx
  matchExpressions:
  - key: tier
    operator: In
    values: [frontend, backend]

Field Selectors

Filter on specific object fields. Support varies by resource — only indexed fields work server-side; others are filtered client-side (all objects fetched, then filtered locally).

kubectl get pods --field-selector status.phase=Running
kubectl get pods --field-selector spec.nodeName=worker-1
kubectl get events --field-selector involvedObject.name=nginx,type=Warning
kubectl get pods --field-selector 'status.phase!=Running,spec.restartPolicy=Always'

OpenAPI v3 and Schema Validation

kube-apiserver publishes per-group OpenAPI v3 schemas at /openapi/v3/apis/{group}/{version}. kubectl uses these for client-side validation. The combined v2 schema is at /openapi/v2.

# Get OpenAPI v3 schema for apps/v1 Deployment
kubectl get --raw '/openapi/v3/apis/apps/v1' | \
  jq '.components.schemas."io.k8s.api.apps.v1.Deployment".properties.spec'

# Validate YAML without applying
kubectl apply --dry-run=client -f deployment.yaml   # client-side only
kubectl apply --dry-run=server -f deployment.yaml   # full server pipeline, no persist

# View structural schema on a CRD
kubectl get crd foos.example.com \
  -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema}' | jq .

CEL Validation in CRDs and ValidatingAdmissionPolicy

CEL (Common Expression Language) rules run in-process in kube-apiserver — no webhook round-trip, sub-millisecond latency. Available in CRD x-kubernetes-validations (GA in 1.30) and ValidatingAdmissionPolicy (GA in 1.30).

CEL in CRDs

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foos.example.com
spec:
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            x-kubernetes-validations:
            - rule: "self.minReplicas <= self.maxReplicas"
              message: "minReplicas must be <= maxReplicas"
            - rule: "self.replicas >= self.minReplicas && self.replicas <= self.maxReplicas"
              message: "replicas must be within [minReplicas, maxReplicas]"
            properties:
              replicas: {type: integer, minimum: 0}
              minReplicas: {type: integer, minimum: 0}
              maxReplicas: {type: integer, minimum: 1}

ValidatingAdmissionPolicy

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-run-as-non-root
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE","UPDATE"]
      resources: ["pods"]
  validations:
  - expression: >
      object.spec.containers.all(c,
        has(c.securityContext) &&
        has(c.securityContext.runAsNonRoot) &&
        c.securityContext.runAsNonRoot == true)
    message: "All containers must set securityContext.runAsNonRoot=true"
  - expression: "object.spec.hostPID == false && object.spec.hostNetwork == false"
    message: "hostPID and hostNetwork must be false"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-run-as-non-root-binding
spec:
  policyName: require-run-as-non-root
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        enforce-security: "true"

CRD Versioning and Conversion Webhooks

CRDs support multiple versions simultaneously. The served flag controls which versions kube-apiserver serves; the storage flag (exactly one version) controls which version is persisted in etcd.

spec:
  versions:
  - name: v1alpha1
    served: true      # /apis/example.com/v1alpha1/... is active
    storage: false    # NOT the etcd storage version
    schema: { ... }
  - name: v1
    served: true
    storage: true     # ALL objects written to etcd as v1
    schema: { ... }
  conversion:
    strategy: Webhook  # None (default) | Webhook
    webhook:
      conversionReviewVersions: ["v1","v1beta1"]
      clientConfig:
        service:
          namespace: default
          name: crd-conversion-webhook
          path: /convert

Hub version pattern: designate one version as the authoritative internal representation. All conversions go through hub: v1alpha1 → hub(v1) → v1alpha1. Avoids O(n²) conversion pairs. The webhook receives a ConversionReview:

# ConversionReview request (sent by apiserver to webhook)
apiVersion: apiextensions.k8s.io/v1
kind: ConversionReview
request:
  uid: "f5cba308-..."
  desiredAPIVersion: example.com/v1
  objects:
  - apiVersion: example.com/v1alpha1
    kind: Foo
    metadata: { name: myfoo }
    spec: { oldField: value }
response:               # webhook returns this
  uid: "f5cba308-..."
  result: {status: Success}
  convertedObjects:
  - apiVersion: example.com/v1
    kind: Foo
    metadata: { name: myfoo }
    spec: { newField: value }  # converted representation

apimachinery Library Layout

Understanding k8s.io/apimachinery is essential for writing controllers and API extensions.

import apierrors "k8s.io/apimachinery/pkg/api/errors"

apierrors.IsNotFound(err)      // 404
apierrors.IsConflict(err)      // 409 (RV mismatch)
apierrors.IsAlreadyExists(err) // 409 (name taken)
apierrors.IsTooManyRequests(err)// 429 (APF throttle)
apierrors.IsServerTimeout(err) // 503
apierrors.IsUnauthorized(err)  // 401
apierrors.IsForbidden(err)     // 403
apierrors.IsInvalid(err)       // 422

type TypeMeta struct {
    Kind       string `json:"kind,omitempty"`
    APIVersion string `json:"apiVersion,omitempty"`
}
type ObjectMeta struct {
    Name, Namespace  string
    UID              types.UID
    ResourceVersion  string
    Generation       int64
    Labels, Annotations map[string]string
    OwnerReferences  []OwnerReference
    Finalizers       []string
}

import "k8s.io/apimachinery/pkg/runtime/schema"

// GVR — used to construct API paths
gvr := schema.GroupVersionResource{
    Group: "apps", Version: "v1",
    Resource: "deployments",
}
// GVK — used in TypeMeta
gvk := schema.GroupVersionKind{
    Group: "apps", Version: "v1",
    Kind: "Deployment",
}
// Convert via RESTMapper
mapper.RESTMappings(gvk.GroupKind(), gvk.Version)

// ListOptions
metav1.ListOptions{
    LabelSelector:   "app=nginx",
    FieldSelector:   "status.phase=Running",
    ResourceVersion: "0",   // from watchCache
    Limit:           500,
}
// Standard condition (use for all status fields)
metav1.Condition{
    Type:    "Ready",
    Status:  metav1.ConditionTrue,
    Reason:  "DeploymentAvailable",
    Message: "3/3 replicas running",
    LastTransitionTime: metav1.Now(),
}

Pagination and Chunking

Large LIST responses should be paginated using limit + continue token to avoid memory pressure on both client and server. All pages share the same resourceVersion snapshot.

# Shell: paginate pods in 100-item chunks
CONTINUE=""
while true; do
  ARGS="--limit=100"
  [ -n "$CONTINUE" ] && ARGS="$ARGS --continue=$CONTINUE"
  RESP=$(kubectl get pods -n default $ARGS -o json)
  echo "$RESP" | jq -r '.items[].metadata.name'
  CONTINUE=$(echo "$RESP" | jq -r '.metadata.continue // empty')
  [ -z "$CONTINUE" ] && break
done

// Go: using client-go pager
import "k8s.io/client-go/tools/pager"

p := pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {
    return client.CoreV1().Pods("").List(ctx, opts)
}))
p.PageSize = 100
err = p.EachListItem(ctx, metav1.ListOptions{}, func(obj runtime.Object) error {
    pod := obj.(*corev1.Pod)
    fmt.Println(pod.Name)
    return nil
})

API Priority and Fairness (APF)

APF replaces the old --max-requests-inflight flag with per-flow fairness queuing. Every request is classified by a FlowSchema and placed in a PriorityLevel queue.

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: my-app-high-priority
spec:
  matchingPrecedence: 200       # lower = matched first
  priorityLevelConfiguration:
    name: workload-high
  rules:
  - subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: my-critical-app
        namespace: default
    resourceRules:
    - verbs: ["*"]
      apiGroups: ["*"]
      resources: ["*"]

# Monitor APF
kubectl get flowschemas
kubectl get prioritylevelconfigurations
kubectl get --raw /metrics | grep 'apiserver_flowcontrol'

# Key metrics:
# apiserver_flowcontrol_current_inqueue_requests{priority_level="workload-high"}
# apiserver_flowcontrol_rejected_requests_total{reason="queue-full"}
# apiserver_flowcontrol_request_wait_duration_seconds

# Diagnose throttling (look for 429 in audit log)
grep '"code":429' /var/log/kubernetes/audit.log | jq '.user.username' | sort | uniq -c

API Troubleshooting

import "k8s.io/client-go/util/retry"

err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
    obj, err := client.Get(ctx, name, metav1.GetOptions{})
    if err != nil { return err }
    obj.Spec.Replicas = &desiredReplicas
    _, err = client.Update(ctx, obj, metav1.UpdateOptions{})
    return err
})

# Check auto-compaction settings
kubectl -n kube-system exec etcd-master -- \
  etcdctl endpoint status --write-out=table

# etcd flags to inspect:
# --auto-compaction-mode=periodic --auto-compaction-retention=5m

kubectl get --raw /metrics | grep -E 'rejected_requests_total|current_inqueue'
# Check which flow schemas the request matches
kubectl get --raw '/apis/flowcontrol.apiserver.k8s.io/v1/flowschemas' | jq '.items[].metadata.name'

# Full error detail
kubectl apply -f resource.yaml 2>&1
# Output includes: "spec.selector: Invalid value ... field is immutable"

# Pre-flight check
kubectl apply --dry-run=server -f resource.yaml

# -v=6: request URL + response code
kubectl get pod nginx -v=6

# -v=8: full request/response headers
kubectl get pod nginx -v=8

# -v=9: full request/response body
kubectl apply -f . -v=9

# Record for analysis
kubectl apply -f . -v=8 2>&1 | tee /tmp/kubectl-debug.log | grep -E 'Request|Response code'

Production Best Practices