Metrics & Prometheus

Prometheus Architecture

Prometheus uses a pull-based model — it scrapes metrics from targets at a configurable interval. Targets expose metrics on an HTTP endpoint (typically /metrics) in the Prometheus exposition format.

TSDB Storage Format

Metric Types

Histogram vs Summary

Copy# Prometheus exposition format examples

# Counter
http_requests_total{method="GET",status="200"} 12345

# Gauge
process_resident_memory_bytes 45678901

# Histogram (automatic _bucket, _count, _sum)
http_request_duration_seconds_bucket{le="0.005"} 100
http_request_duration_seconds_bucket{le="0.01"}  200
http_request_duration_seconds_bucket{le="0.025"} 350
http_request_duration_seconds_bucket{le="0.05"}  400
http_request_duration_seconds_bucket{le="+Inf"}  500
http_request_duration_seconds_count              500
http_request_duration_seconds_sum                12.345

Scrape Configuration

Copy# prometheus.yaml — scrape configuration
global:
  scrape_interval: 15s         # Default scrape interval
  evaluation_interval: 15s     # Rules evaluation interval
  scrape_timeout: 10s

scrape_configs:

# Scrape Kubernetes API server
- job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names: [default]
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

# Scrape all pods with annotation prometheus.io/scrape: "true"
- job_name: kubernetes-pods
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod

Prometheus Operator

The Prometheus Operator extends Kubernetes with CRDs that let you declare Prometheus instances and scrape targets as Kubernetes objects, enabling GitOps-compatible monitoring configuration.

Core CRDs

ServiceMonitor

Copy# ServiceMonitor: tells Prometheus Operator which Services to scrape
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: production
  labels:
    team: backend    # Must match Prometheus.spec.serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: myapp      # Selects Services with this label
  namespaceSelector:
    matchNames: [production]
  endpoints:
  - port: metrics          # Named port on the Service
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s
    scheme: http
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node
    metricRelabelings:
    # Drop high-cardinality metrics at ingest time
    - sourceLabels: [__name__]
      regex: go_gc_.*
      action: drop

PrometheusRule

CopyapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-rules
  namespace: production
  labels:
    team: backend
spec:
  groups:
  - name: myapp.recording
    interval: 30s
    rules:
    - record: job:http_requests:rate5m
      expr: sum by (job) (rate(http_requests_total[5m]))
  - name: myapp.alerts
    rules:
    - alert: HighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate on {{ $labels.job }}"
        description: "Error rate is {{ $value | humanizePercentage }}"

kube-prometheus-stack

The kube-prometheus-stack Helm chart is the standard way to deploy the full Prometheus monitoring stack with pre-built dashboards, recording rules, and alert rules for Kubernetes.

Copy# Add Helm repo and install
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values kps-values.yaml

Copy# kps-values.yaml — production configuration
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 50GB
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 100Gi
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        memory: 4Gi
    # Accept ServiceMonitors from all namespaces
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    # TSDB compaction and query tuning
    tsdb:
      outOfOrderTimeWindow: 10m   # Accept late-arriving samples
    walCompression: true

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          resources:
            requests:
              storage: 10Gi

grafana:
  adminPassword: changeme-use-secret
  persistence:
    enabled: true
    size: 10Gi

nodeExporter:
  enabled: true

kubeStateMetrics:
  enabled: true

Key Kubernetes Metrics

Container Resources (cAdvisor)

Kubernetes State (kube-state-metrics)

API Server

PromQL

PromQL (Prometheus Query Language) is a functional query language for time series data. It operates on selectors, range vectors, and a rich set of built-in functions.

Selectors and Labels

Copy# Instant vector: current value
http_requests_total

# Label equality filter
http_requests_total{job="myapp", status="200"}

# Label regex match
http_requests_total{status=~"5.."}        # 5xx errors
http_requests_total{status!~"2.."}        # Not 2xx

# Range vector: samples over a time window
http_requests_total[5m]                   # Last 5 minutes of samples

# Offset: look back in time
http_requests_total offset 1h             # Value 1 hour ago

# Subquery: range query over instant expression
rate(http_requests_total[5m])[1h:5m]      # rate over 5m, sampled every 5m for 1h

Essential Functions

Copy# rate(): per-second rate of increase of a counter (avg over window)
# Use for alerting — smooths out spikes
rate(http_requests_total[5m])

# irate(): instant rate — uses last two samples only
# More responsive to spikes; noisy for alerts
irate(http_requests_total[5m])

# increase(): total increase over window (rate * window seconds)
increase(http_requests_total[1h])    # Approx requests in last hour

# histogram_quantile(): calculate percentile from histogram
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# sum by: aggregate across labels
sum by (namespace, pod) (rate(container_cpu_usage_seconds_total[5m]))

# without: aggregate, dropping specified labels
sum without (instance, pod) (rate(http_requests_total[5m]))

# topk: top N time series by value
topk(10, rate(http_requests_total[5m]))

# absent: alert when metric is missing (target down)
absent(up{job="myapp"} == 1)

# predict_linear: forecast gauge value
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
# Will disk be full in 4 hours?

Common Production Queries

Copy# Container CPU utilization % (vs request)
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
) /
sum by (namespace, pod, container) (
  kube_pod_container_resource_requests{resource="cpu", container!=""}
) * 100

# Container memory utilization % (vs limit)
sum by (namespace, pod, container) (
  container_memory_working_set_bytes{container!=""}
) /
sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="memory", container!=""}
) * 100

# Error rate for HTTP services
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))

# p99 latency across all instances of a service
histogram_quantile(0.99,
  sum by (job, le) (
    rate(http_request_duration_seconds_bucket{job="myapp"}[5m])
  )
)

# Pods NOT running (pending/failed/unknown)
count by (namespace) (
  kube_pod_status_phase{phase!="Running", phase!="Succeeded"}
)

# Nodes with memory pressure
kube_node_status_condition{condition="MemoryPressure", status="true"} == 1

Recording Rules

Recording rules pre-compute expensive PromQL expressions and store the result as a new time series. This dramatically speeds up dashboard load times and reduces query-time CPU load.

Copy# PrometheusRule with recording rules
groups:
- name: kubernetes-resources.recording
  interval: 30s
  rules:

  # CPU utilization ratio (pre-computed for dashboard)
  - record: namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
    expr: |
      sum by (namespace, pod, container) (
        irate(container_cpu_usage_seconds_total{job="kubelet", container!=""}[5m])
      )

  # Request rate per job
  - record: job:http_requests_total:rate5m
    expr: |
      sum by (job, status) (rate(http_requests_total[5m]))

  # Error ratio — used by SLO burn rate alert
  - record: job:http_request_errors:ratio_rate5m
    expr: |
      sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum by (job) (rate(http_requests_total[5m]))

  # p99 latency recording rule
  - record: job:http_request_duration_seconds:histogram_quantile99_rate5m
    expr: |
      histogram_quantile(0.99,
        sum by (job, le) (
          rate(http_request_duration_seconds_bucket[5m])
        )
      )

Cardinality Management

Cardinality is the number of unique time series in Prometheus. It is the primary driver of Prometheus memory usage and query performance. High cardinality is the most common operational problem.

Cardinality Analysis

Copy# Check total time series count
prometheus_tsdb_head_series

# Top 10 jobs by series count (PromQL)
topk(10,
  count by (job) ({__name__=~".+"})
)

# Top 10 metrics by series count
topk(10,
  count by (__name__) ({__name__=~".+"})
)

# Series count for a specific metric
count(http_requests_total)

# cardinality API endpoint (Prometheus 2.14+)
curl http://prometheus:9090/api/v1/status/tsdb | jq .data.headStats

# Detailed cardinality analysis
curl http://prometheus:9090/api/v1/status/tsdb?limit=20 | \
  jq '.data.labelValueCountByLabelName | sort_by(.labelValueCount) | reverse | .[0:10]'

Reducing Cardinality

Copy# Drop high-cardinality labels via metric relabeling in ServiceMonitor
metricRelabelings:
# Drop pod-template-hash label (changes every deployment, creates new series)
- action: labeldrop
  regex: pod_template_hash

# Drop entire metrics that are not needed
- sourceLabels: [__name__]
  regex: go_gc_duration_seconds.*
  action: drop

# Replace high-cardinality URL paths with bucketed versions
- sourceLabels: [path]
  regex: /api/users/[0-9]+
  targetLabel: path
  replacement: /api/users/:id

Copy# Prometheus per-scrape cardinality limit (prevent a single job from exploding)
scrape_configs:
- job_name: myapp
  sample_limit: 10000    # Max samples per scrape (drops the scrape if exceeded)
  label_limit: 64        # Max labels per sample
  label_name_length_limit: 128
  label_value_length_limit: 256

Long-Term Storage: Thanos & Mimir

Prometheus stores data locally with a default retention of 15 days. For long-term storage, multi-cluster aggregation, and high availability, use Thanos or Grafana Mimir.

Thanos Architecture

Copy# Thanos Sidecar: prometheus.yaml addition
# Add to kube-prometheus-stack values:
prometheus:
  prometheusSpec:
    thanos:
      image: quay.io/thanos/thanos:v0.35.0
      objectStorageConfig:
        existingSecret:
          name: thanos-objstore-secret
          key: objstore.yml
---
# objstore.yml (stored as Secret)
type: S3
config:
  bucket: my-thanos-bucket
  endpoint: s3.amazonaws.com
  region: us-east-1
  aws_sdk_auth: true    # Use IRSA

Grafana Mimir

Grafana Mimir is a horizontally scalable, multi-tenant Prometheus-compatible backend. It is the successor to Cortex and replaces the Thanos Sidecar + Store Gateway pattern with a unified write path.

Remote Write & Federation

Copy# Remote write: ship metrics to external backend
remote_write:
- url: https://mimir.example.com/api/v1/push
  headers:
    X-Scope-OrgID: production
  remote_timeout: 30s
  queue_config:
    capacity: 10000
    max_shards: 30
    min_shards: 5
    max_samples_per_send: 5000
    batch_send_deadline: 5s
  write_relabel_configs:
  # Only send metrics with specific labels to reduce egress
  - sourceLabels: [__name__]
    regex: job:.*|kube_.*|node_.*
    action: keep

Copy# Federation: aggregate subset of metrics from child Prometheus
scrape_configs:
- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    match[]:
    - '{__name__=~"job:.*"}'        # Only recording rules
    - up
  static_configs:
  - targets:
    - prometheus-cluster-a:9090
    - prometheus-cluster-b:9090

Metrics, Alerts & Runbooks

Key Prometheus Self-Metrics

Alerts

Copy# Alert: Prometheus target down
- alert: PrometheusTargetDown
  expr: up == 0
  for: 5m
  annotations:
    summary: "Scrape target {{ $labels.job }} / {{ $labels.instance }} is down"

# Alert: Prometheus cardinality too high
- alert: PrometheusHighCardinality
  expr: prometheus_tsdb_head_series > 2000000
  for: 15m
  annotations:
    summary: "Prometheus has >2M active series — memory pressure risk"

# Alert: Prometheus rule evaluation slow
- alert: PrometheusRuleEvaluationSlow
  expr: prometheus_rule_evaluation_duration_seconds{quantile="0.9"} > 1
  for: 10m
  annotations:
    summary: "Rule evaluation p90 > 1s — rules may miss evaluation windows"

# Alert: Remote write queue filling
- alert: PrometheusRemoteWriteQueueFull
  expr: |
    prometheus_remote_storage_shard_capacity -
    prometheus_remote_storage_pending_examples < 100
  for: 5m
  annotations:
    summary: "Remote write queue almost full — samples may be dropped"

Runbooks

Best Practices