Local Development

Overview

Local Kubernetes development solves one problem: make the inner loop fast. The inner loop is the cycle a developer runs dozens of times per hour — edit code, see it running, verify behaviour. Every second of friction in that loop compounds across a team.

Inner loop (target: < 5 seconds end-to-end)
  Edit code → rebuild image → redeploy → see logs → repeat

Outer loop (CI/CD, runs on push)
  git push → build → scan → sign → deploy to staging → promote to prod

This page covers local cluster options, hot-reload tools (Tilt, Skaffold), devcontainers, and the workflow patterns that keep developer laptops in sync with production behaviour.

Local Cluster Options

Comparison Table

kind — Kubernetes in Docker

kind runs K8s nodes as Docker containers. It uses kubeadm internally, so its behaviour is the closest to production of any local option.

# Install kind
brew install kind   # macOS
# or
curl -Lo /usr/local/bin/kind \
  https://kind.sigs.k8s.io/dl/v0.25.0/kind-linux-amd64 && \
  chmod +x /usr/local/bin/kind

# Create a single-node cluster
kind create cluster --name dev

# Create a multi-node cluster that mirrors production topology
cat <<'EOF' > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  labels:
    topology.kubernetes.io/zone: us-east-1a
- role: worker
  labels:
    topology.kubernetes.io/zone: us-east-1b
- role: worker
  labels:
    topology.kubernetes.io/zone: us-east-1c
EOF
kind create cluster --name dev --config kind-config.yaml

# Load a locally-built image into the cluster (avoids registry push)
docker build -t payments-api:dev .
kind load docker-image payments-api:dev --name dev

# Delete cluster
kind delete cluster --name dev

k3d — k3s in Docker

k3d wraps k3s (a lightweight K8s distribution) in Docker containers. Fastest startup, lowest RAM, good enough for most application-level dev.

# Install k3d
brew install k3d

# Create cluster with a local registry (avoids kind load for every build)
k3d registry create dev-registry --port 5001

k3d cluster create dev \
  --registry-use k3d-dev-registry:5001 \
  --agents 2 \
  --k3s-arg "--disable=traefik@server:0"   # disable built-in ingress

# Build and push to local registry — k3d picks it up automatically
docker build -t localhost:5001/payments-api:dev .
docker push localhost:5001/payments-api:dev

# Use in pod spec:
#   image: k3d-dev-registry:5001/payments-api:dev

vCluster — Virtual Clusters for Teams

vCluster creates a fully functional K8s API server inside a namespace of a host cluster. Each developer or squad gets their own isolated API server with no interference.

# Install vcluster CLI
brew install loft-sh/tap/vcluster

# Create a virtual cluster in the shared dev cluster
vcluster create my-dev-cluster --namespace my-dev-ns

# Connect to it (sets kubeconfig context)
vcluster connect my-dev-cluster --namespace my-dev-ns

# Work normally — all kubectl commands go to the virtual cluster
kubectl get pods   # isolated namespace view

# Disconnect and switch back to host
vcluster disconnect

Tilt — Hot Reload for Kubernetes

Tilt watches your source files, rebuilds only what changed, and syncs updates into running containers — often without a pod restart. It provides a live web UI showing all resource statuses, logs, and build output.

Installation

# macOS
brew install tilt-dev/tap/tilt

# Linux
curl -fsSL https://raw.githubusercontent.com/tilt-dev/tilt/master/scripts/install.sh | bash

Basic Tiltfile

# Tiltfile (Python-like DSL)

# Build the Docker image
docker_build(
    'payments-api',
    '.',                          # build context
    dockerfile='Dockerfile',
    # Sync source files into the container without rebuilding the image
    # Works when the container runs a file-watching process (nodemon, air, watchexec)
    live_update=[
        sync('./src', '/app/src'),          # sync Go/Node source
        run('go build -o /app/server ./cmd/server', trigger=['./src']),
    ]
)

# Apply Kubernetes manifests
k8s_yaml('k8s/deployment.yaml')
k8s_yaml('k8s/service.yaml')

# Group resources and configure port-forwarding
k8s_resource(
    'payments-api',
    port_forwards=['8080:8080', '6060:6060'],  # app + pprof
    labels=['backend']
)

# Dependencies — start postgres before payments-api
k8s_resource('postgres', labels=['infra'])
k8s_resource('payments-api', resource_deps=['postgres'])

Tiltfile with Helm

# Use Helm chart for deployment
load('ext://helm_resource', 'helm_resource', 'helm_repo')

docker_build('payments-api', '.', live_update=[
    sync('./src', '/app/src'),
])

helm_resource(
    'payments-api',
    chart='./charts/payments-api',
    namespace='dev',
    image_deps=['payments-api'],
    image_keys=[('image.repository', 'image.tag')],
    flags=['--values=./charts/payments-api/values-local.yaml'],
)

Tiltfile with Kustomize

docker_build('payments-api', '.')

# Apply kustomize overlay for local development
k8s_yaml(kustomize('./k8s/overlays/local'))

k8s_resource('payments-api', port_forwards='8080:8080')

Live Update Pattern — Go (air)

# Dockerfile.dev — for local development only
FROM golang:1.23-alpine
WORKDIR /app
RUN go install github.com/air-verse/air@latest
COPY go.mod go.sum ./
RUN go mod download
# Source copied in by Tilt live_update sync — no COPY . . here
CMD ["air", "-c", ".air.toml"]

# Tiltfile — sync changes, trigger air recompile
docker_build(
    'payments-api',
    '.',
    dockerfile='Dockerfile.dev',
    live_update=[
        sync('.', '/app'),
        run('cd /app && go mod tidy', trigger=['go.mod', 'go.sum']),
    ]
)

Live Update Pattern — Node.js

docker_build(
    'frontend',
    '.',
    dockerfile='Dockerfile.dev',
    live_update=[
        fall_back_on(['package.json', 'package-lock.json']),  # full rebuild if deps change
        sync('./src', '/app/src'),
        sync('./public', '/app/public'),
    ]
)

Tilt Web UI

# Start Tilt — opens web UI at http://localhost:10350
tilt up

# Run in CI mode (no UI, exit on error)
tilt ci

# Tear down all resources
tilt down

Skaffold — Build/Deploy Pipeline for Local Dev

Skaffold is more pipeline-oriented than Tilt. It defines explicit build → test → deploy stages and integrates with multiple builders (Docker, Buildpacks, Jib for Java) and deployers (kubectl, Helm, Kustomize).

Installation

brew install skaffold
# or
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64
chmod +x skaffold && mv skaffold /usr/local/bin

skaffold.yaml

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: payments-api

build:
  artifacts:
  - image: payments-api
    docker:
      dockerfile: Dockerfile
      buildArgs:
        GO_VERSION: "1.23"
    sync:
      # File sync — push changes without rebuilding the image
      manual:
      - src: "src/**/*.go"
        dest: /app/src
  local:
    push: false       # don't push to registry for local dev
    useBuildkit: true

test:
- image: payments-api
  custom:
  - command: go test ./...
    timeoutSeconds: 120

deploy:
  kubectl:
    manifests:
    - k8s/*.yaml

# Profiles for different environments
profiles:
- name: staging
  build:
    local:
      push: true
  deploy:
    helm:
      releases:
      - name: payments-api
        chartPath: charts/payments-api
        valuesFiles: [charts/payments-api/values-staging.yaml]

- name: production
  build:
    googleCloudBuild:
      projectId: my-gcp-project
  deploy:
    helm:
      releases:
      - name: payments-api
        chartPath: charts/payments-api
        valuesFiles: [charts/payments-api/values-production.yaml]

Skaffold Commands

# Dev mode — watch, rebuild, redeploy on change
skaffold dev

# Dev mode with a specific profile
skaffold dev --profile staging

# One-shot build and deploy (for CI)
skaffold run

# Build only (produce image, don't deploy)
skaffold build --file-output=build.json

# Deploy only using a previous build output
skaffold deploy --build-artifacts=build.json

# Debug mode — auto-configure debugger ports per language
# Java: JDWP port 5005; Node: --inspect port 9229; Go: dlv port 56268
skaffold debug

# Render manifests without deploying (dry-run)
skaffold render

Tilt vs Skaffold — Decision Guide

devcontainer — Reproducible Dev Environments

A devcontainer (VS Code Dev Containers / GitHub Codespaces) packages the entire development environment — tools, SDKs, shell config — into a Docker image. Every developer gets identical tooling.

.devcontainer/devcontainer.json

{
  "name": "Kubernetes Dev",
  "image": "mcr.microsoft.com/devcontainers/go:1.23",
  "features": {
    "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": {
      "version": "latest",
      "helm": "latest",
      "minikube": "none"
    },
    "ghcr.io/devcontainers/features/docker-in-docker:2": {},
    "ghcr.io/devcontainers/features/github-cli:1": {}
  },
  "postCreateCommand": "make install-tools",
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-kubernetes-tools.vscode-kubernetes-tools",
        "redhat.vscode-yaml",
        "golang.go",
        "ms-azuretools.vscode-docker"
      ],
      "settings": {
        "go.toolsManagement.autoUpdate": true
      }
    }
  },
  "mounts": [
    "source=${localWorkspaceFolder},target=/workspace,type=bind"
  ],
  "remoteUser": "vscode",
  "forwardPorts": [8080, 10350]
}

Makefile — Common Dev Tasks

.PHONY: install-tools cluster up down logs test

# Install local tooling
install-tools:
	go install github.com/air-verse/air@latest
	go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
	brew install tilt-dev/tap/tilt kind kubectx stern

# Create local kind cluster
cluster:
	kind create cluster --name dev --config hack/kind-config.yaml
	kubectl apply -f hack/local-registry-hosting.yaml

# Start Tilt (inner loop)
up:
	tilt up

# Tear down
down:
	tilt down
	kind delete cluster --name dev

# Stream logs from all services
logs:
	stern -n dev .

# Run tests
test:
	go test ./... -race -cover

Local Secrets Management

Never commit secrets. For local dev, use one of these patterns:

Pattern 1: .env file + envsubst

# .env.local (git-ignored)
DB_PASSWORD=localpassword
API_KEY=localkey

# Apply with envsubst
export $(cat .env.local | xargs)
envsubst < k8s/secret.yaml.template | kubectl apply -f -

Pattern 2: Doppler / 1Password CLI

# Doppler injects secrets as env vars at process start
doppler run -- tilt up

# 1Password CLI — inject secrets into kubeconfig or manifests
op run --env-file=.env.template -- tilt up

Pattern 3: Local Vault dev mode

# Start Vault in dev mode (in-memory, for local dev only)
vault server -dev -dev-root-token-id=root &

# Configure Vault secrets
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root
vault kv put secret/payments db_password=localpassword

# Use External Secrets Operator (local) pointing at local Vault
# — same ESO manifests as production, just different Vault address

Debugging Running Pods Locally

# Attach a debugger to a running Go process (using delve)
# Requires delve to be in the container and GOMAXPROCS-aware
kubectl port-forward pod/payments-api-xxx 2345:2345

# VS Code launch.json for remote delve
# {
#   "type": "go",
#   "request": "attach",
#   "mode": "remote",
#   "remotePath": "/app",
#   "port": 2345,
#   "host": "127.0.0.1"
# }

# Drop into a running container shell
kubectl exec -it deploy/payments-api -- /bin/sh

# Add an ephemeral debug container (no shell in distroless images)
kubectl debug -it deploy/payments-api \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -- bash

# Stream logs from all pods matching a label
stern -l app=payments-api -n dev --since=5m

# Port-forward multiple services at once (use tmux or background)
kubectl port-forward svc/payments-api 8080:8080 -n dev &
kubectl port-forward svc/postgres 5432:5432 -n dev &
kubectl port-forward svc/redis 6379:6379 -n dev &

Local Observability

# Install kube-prometheus-stack in local kind cluster (reduced resource footprint)
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.retention=2h \
  --set prometheus.prometheusSpec.resources.requests.memory=256Mi \
  --set alertmanager.enabled=false \
  --set grafana.adminPassword=admin

# Port-forward Grafana
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

# Quick metrics check without Prometheus — use kubectl top
kubectl top pods -n dev
kubectl top nodes

Common Issues and Fixes

Related

• CI/CD Pipelines — outer loop: building, scanning, deploying via CI
• Helm — packaging services for deployment
• Testing Strategies — running tests against local cluster
• 09 — Performance Tuning — understanding resource limits set during local dev