Observability Overview

Why Observability

Monitoring asks "is this thing healthy?" — a binary answer based on known failure modes. Observability asks "why is this behaving this way?" — allowing exploration of unknown unknowns without deploying new instrumentation.

The Three Pillars

Signal Comparison

The Fourth Signal: Events

Kubernetes Events are a native signal type — structured records of state changes in the cluster (pod scheduled, container crashed, image pulled, node pressure). They are often overlooked but are invaluable for correlating infrastructure changes with application symptoms. See Kubernetes Events for the full treatment.

Kubernetes Observability Layers

Observability in Kubernetes spans four distinct layers, each with different data sources, tooling, and ownership.

Key Data Sources per Layer

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for vendor-neutral instrumentation and telemetry collection. It unifies metrics, logs, and traces under a single SDK, wire protocol (OTLP), and collection pipeline (OTel Collector).

OTel Collector Pipeline

Copy# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: otel-collector
        static_configs:
        - targets: [localhost:8888]
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      filesystem: {}

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  resource:
    attributes:
    - key: k8s.cluster.name
      value: production
      action: upsert

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, batch, resource]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [loki]

OTel Operator for Kubernetes

Copy# Install OTel Operator
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry-operator-system \
  --create-namespace

# OpenTelemetryCollector CRD — operator manages collector lifecycle
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
spec:
  mode: daemonset   # daemonset | deployment | sidecar | statefulset
  config: |
    # inline OTel Collector config (see above)

Copy# Instrumentation CRD: auto-instrument pods without code changes
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: auto-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators: [tracecontext, baggage, b3]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"   # 10% sampling
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.32.0
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.41.1
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.41b0
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:v0.9.0-alpha

# Opt pod into auto-instrumentation via annotation:
# instrumentation.opentelemetry.io/inject-java: "true"
# instrumentation.opentelemetry.io/inject-nodejs: "true"

Signal Correlation

The power of observability comes from correlating signals — jumping from a metric alert to the relevant logs and traces for the same time window and request.

Exemplars: Linking Metrics to Traces

Copy# Prometheus exemplar: attach trace_id to histogram observation
# (Go example with prometheus/client_golang)
httpDuration.With(labels).ObserveWithExemplar(
    duration,
    prometheus.Labels{"traceID": traceID},
)

# Enable exemplar storage in Prometheus
# prometheus.yaml
storage:
  exemplars:
    max-exemplars: 100000

Trace ID in Logs

Copy# Inject trace ID into structured logs (Go + Zap + OTel)
span := trace.SpanFromContext(ctx)
logger.Info("processing request",
    zap.String("trace_id", span.SpanContext().TraceID().String()),
    zap.String("span_id",  span.SpanContext().SpanID().String()),
    zap.String("service",  "payment-svc"),
)

# Log output (JSON):
{
  "level": "info",
  "ts": "2024-01-15T14:32:15.123Z",
  "msg": "processing request",
  "trace_id": "abc123def456...",
  "span_id": "0102030405060708",
  "service": "payment-svc"
}

# Loki query: find logs for a specific trace
{namespace="production"} | json | trace_id = "abc123def456..."

Tooling Landscape

CNCF Open Source Stack (LGTM)

Stack Comparison

Instrumentation Strategy

Auto-Instrumentation vs Manual Instrumentation

What to Instrument First

SLOs, SLIs, and Error Budgets

SLOs (Service Level Objectives) transform raw metrics into business-facing reliability targets. They are the bridge between engineering observability and product/business requirements.

Copy# Multi-window burn rate alert (Google SRE approach)
# Fires when error budget is being consumed faster than sustainable

# Short window (fast detection) + long window (sustained burn)
- alert: SLOBurnRateHigh
  expr: |
    (
      # 1h burn rate > 14.4x (uses 2% budget in 1h at this rate)
      job:slo_errors:rate1h{job="payment-svc"} / 0.001 > 14.4
    ) and (
      # 5m burn rate > 14.4x (ensures it's sustained)
      job:slo_errors:rate5m{job="payment-svc"} / 0.001 > 14.4
    )
  severity: page
  annotations:
    summary: "SLO burn rate critical — paging"

Observability Cost Management

Observability infrastructure can become one of the largest operational costs in a Kubernetes platform. Understanding the cost drivers enables targeted optimization.

Cost Drivers and Mitigations

Section Guide