Audit Logging

Audit Logging Overview

Kubernetes audit logging records every request processed by the API server — who made the request, what they requested, and what the outcome was. It is the authoritative trail for security investigation, compliance, and forensics.

What Audit Logs Answer

Policy Levels & Stages

Audit Levels

Each audit rule assigns a level that controls how much detail is recorded. Higher levels capture more data but increase log volume and API server overhead.

Audit Stages

Writing Audit Policies

The audit policy is a YAML file passed to the API server via --audit-policy-file. Rules are evaluated in order; the first matching rule determines the level. If no rule matches, the request is not logged.

Minimal Policy (Baseline)

CopyapiVersion: audit.k8s.io/v1
kind: Policy

# Omit ResponseStarted for watch-type long-running requests
omitStages:
- RequestReceived

rules:
# Rule 1: Don't log read-only requests from system components
- level: None
  users:
  - system:kube-scheduler
  - system:kube-proxy
  - system:apiserver
  - system:kube-controller-manager
  - system:serviceaccount:kube-system:endpoint-controller
  verbs: [get, watch, list]

# Rule 2: Don't log health/readiness probes
- level: None
  nonResourceURLs:
  - /healthz*
  - /readyz*
  - /livez*
  - /version
  - /swagger*
  - /openapi*

# Rule 3: Don't log metrics scrape
- level: None
  nonResourceURLs:
  - /metrics
  - /metrics/cadvisor

# Rule 4: Full detail for secrets (no response body — value excluded)
- level: Request
  resources:
  - group: ""
    resources: [secrets, configmaps, serviceaccounts/token]

# Rule 5: Full detail for RBAC changes
- level: RequestResponse
  resources:
  - group: rbac.authorization.k8s.io
    resources: [roles, clusterroles, rolebindings, clusterrolebindings]

# Rule 6: Log exec/attach/portforward at Request level (captures command args)
- level: Request
  resources:
  - group: ""
    resources: [pods/exec, pods/attach, pods/portforward]

# Rule 7: Metadata for most other resources
- level: Metadata
  resources:
  - group: ""
  - group: apps
  - group: batch
  - group: networking.k8s.io
  - group: policy
  - group: storage.k8s.io

Production Policy (Tiered)

A production policy uses a tiered approach: suppress noise aggressively, capture sensitive operations in full, and use Metadata as the safe default for everything else.

CopyapiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
omitManagedFields: true   # Removes managedFields from request/response bodies

rules:

# ── TIER 0: Complete suppression (high volume, no security value) ────

- level: None
  userGroups: [system:nodes]
  verbs: [get, watch, list]
  resources:
  - group: ""
    resources: [endpoints, services, pods, nodes]

- level: None
  nonResourceURLs: [/healthz*, /readyz*, /livez*, /version, /metrics*]

# Suppress leader election coordination (very high volume, no security value)
- level: None
  resources:
  - group: coordination.k8s.io
    resources: [leases]

# Suppress event updates (informational only, not security relevant)
- level: None
  resources:
  - group: ""
    resources: [events]

# ── TIER 1: RequestResponse — highest security value ─────────────────

# RBAC mutations — full request+response for privilege escalation detection
- level: RequestResponse
  verbs: [create, update, patch, delete]
  resources:
  - group: rbac.authorization.k8s.io
    resources:
    - roles
    - clusterroles
    - rolebindings
    - clusterrolebindings

# ServiceAccount token creation and SA mutations
- level: RequestResponse
  resources:
  - group: ""
    resources: [serviceaccounts, serviceaccounts/token]

# Pod exec/attach — captures command run (command in URI params)
- level: Request
  resources:
  - group: ""
    resources: [pods/exec, pods/attach, pods/portforward]

# ── TIER 2: Request — important writes ───────────────────────────────

# Secret reads and writes (NOT RequestResponse — would log secret values)
- level: Request
  verbs: [get, list, create, update, patch, delete]
  resources:
  - group: ""
    resources: [secrets]

# Admission webhook configurations — changes here affect all cluster security
- level: RequestResponse
  resources:
  - group: admissionregistration.k8s.io
    resources: [mutatingwebhookconfigurations, validatingwebhookconfigurations]

# ── TIER 3: Metadata — default for everything else ───────────────────
- level: Metadata

API Server Flags

Copy# kube-apiserver flags for audit logging
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=30          # Days to retain rotated log files
--audit-log-maxbackup=10       # Max number of old log files to keep
--audit-log-maxsize=100        # Max size in MB before rotation
--audit-log-compress           # Compress rotated files with gzip
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-format=json        # json (default) or legacy (text)

Audit Backends

Log Backend

The log backend writes audit events to a file on the API server node in JSON Lines format (one JSON object per line). This is the simplest setup and suitable for environments with a log collection agent (Fluentd, Fluent Bit, Filebeat) on the node.

Webhook Backend

The webhook backend sends audit events to an external HTTP endpoint in batches. Used when you want real-time delivery to a SIEM or an audit aggregator without first writing to disk.

Copy# webhook-audit-config.yaml — kubeconfig-format endpoint configuration
apiVersion: v1
kind: Config
clusters:
- name: audit-webhook
  cluster:
    server: https://audit-sink.internal:8888/audit
    certificate-authority: /etc/kubernetes/audit-webhook-ca.crt
contexts:
- name: webhook
  context:
    cluster: audit-webhook
    user: ""
current-context: webhook

Copy# API server flags for webhook backend
--audit-webhook-config-file=/etc/kubernetes/webhook-audit-config.yaml
--audit-webhook-mode=batch          # batch (default) or blocking
--audit-webhook-batch-max-size=400  # Events per batch
--audit-webhook-batch-max-wait=30s  # Max time before sending incomplete batch
--audit-webhook-initial-backoff=10s # Initial retry delay
--audit-webhook-truncate-enabled    # Truncate oversized events instead of dropping
--audit-webhook-truncate-max-event-size=10485760  # 10MB max event

Audit Event Format

Each audit event is a JSON object with a well-defined schema. Understanding the fields is essential for writing effective SIEM queries and Falco rules.

Copy{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",

  // Unique ID for this event (UUID)
  "auditID": "f23a4b56-1234-5678-abcd-000000000001",

  // Which stage this event was emitted from
  "stage": "ResponseComplete",

  // The full request URI
  "requestURI": "/api/v1/namespaces/production/secrets/db-password",

  // HTTP verb
  "verb": "get",

  // Authenticated user information
  "user": {
    "username": "system:serviceaccount:production:myapp",
    "uid": "abc123",
    "groups": ["system:serviceaccounts", "system:serviceaccounts:production"]
  },

  // If request was made via impersonation
  "impersonatedUser": {
    "username": "admin@example.com"
  },

  // Source IPs (first is original, rest are proxies)
  "sourceIPs": ["10.244.1.5", "192.168.1.100"],

  // User-Agent of the client
  "userAgent": "kubectl/v1.29.0 (linux/amd64)",

  // What resource was accessed
  "objectRef": {
    "resource": "secrets",
    "namespace": "production",
    "name": "db-password",
    "apiVersion": "v1"
  },

  // HTTP response code
  "responseStatus": {
    "code": 200
  },

  // Request body (if level=Request or RequestResponse)
  "requestObject": { "...": "..." },

  // Response body (if level=RequestResponse)
  "responseObject": { "...": "..." },

  // Timestamps
  "requestReceivedTimestamp": "2024-01-15T10:30:00.000000Z",
  "stageTimestamp": "2024-01-15T10:30:00.005000Z",

  // Annotations added by admission plugins
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding..."
  }
}

Noise Reduction

An untuned audit policy on a production cluster can generate hundreds of megabytes of logs per hour. Most of this is noise. The goal is to capture high-value events without drowning the pipeline.

High-Volume Low-Value Sources

Volume Estimation

Copy# Estimate audit log volume before deploying (dry-run with --audit-log-path=-)
# Then count events per second during normal operation:

# Count events per minute from audit log
tail -f /var/log/kubernetes/audit/audit.log | \
  jq -r '.stageTimestamp' | \
  awk -F: '{print $1":"$2}' | uniq -c

# Top users by event count (last 1000 events)
tail -1000 /var/log/kubernetes/audit/audit.log | \
  jq -r '.user.username' | sort | uniq -c | sort -rn | head -20

# Top resources by event count
tail -1000 /var/log/kubernetes/audit/audit.log | \
  jq -r '.objectRef.resource // "non-resource"' | sort | uniq -c | sort -rn | head -20

SIEM Integration

Fluent Bit Configuration for Audit Logs

Copy# fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Parsers_File  parsers.conf

    # Read audit log file — must be on control plane node
    [INPUT]
        Name              tail
        Tag               kube.audit
        Path              /var/log/kubernetes/audit/audit.log
        Parser            json
        DB                /var/log/flb_kube_audit.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    # Add cluster metadata
    [FILTER]
        Name  record_modifier
        Match kube.audit
        Record cluster ${CLUSTER_NAME}
        Record environment ${ENVIRONMENT}

    # Forward to Elasticsearch
    [OUTPUT]
        Name  es
        Match kube.audit
        Host  elasticsearch.logging.svc.cluster.local
        Port  9200
        Index k8s-audit
        Type  _doc
        Logstash_Format On
        Logstash_Prefix k8s-audit

Elasticsearch Query Examples

Copy# Elasticsearch: find all secret reads in the last hour
GET k8s-audit-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "objectRef.resource": "secrets" } },
        { "term": { "verb": "get" } },
        { "range": { "stageTimestamp": { "gte": "now-1h" } } }
      ],
      "must_not": [
        { "term": { "user.username": "system:serviceaccount:kube-system:cert-manager" } }
      ]
    }
  }
}

Forensic Query Patterns

These jq patterns work against raw audit log files and translate directly to SIEM queries.

Secret Access Investigation

Copy# All users who read any secret in the last 24h
cat audit.log | jq -r '
  select(.objectRef.resource == "secrets") |
  select(.verb == "get") |
  select(.responseStatus.code == 200) |
  [.stageTimestamp, .user.username, .objectRef.namespace, .objectRef.name] |
  @tsv
'

# Secrets read by service accounts outside their own namespace
cat audit.log | jq -r '
  select(.objectRef.resource == "secrets") |
  select(.verb == "get") |
  select(.user.username | startswith("system:serviceaccount:")) |
  select(
    (.user.username | split(":")[3]) !=
    (.objectRef.namespace // "cluster-scoped")
  ) |
  [.stageTimestamp, .user.username, .objectRef.namespace, .objectRef.name] |
  @tsv
'

Privilege Escalation Detection

Copy# New ClusterRoleBindings created (potential privilege escalation)
cat audit.log | jq -r '
  select(.objectRef.resource == "clusterrolebindings") |
  select(.verb == "create") |
  select(.responseStatus.code == 201) |
  [.stageTimestamp, .user.username, .objectRef.name,
   (.requestObject.roleRef.name // "unknown")] |
  @tsv
'

# Any binding to cluster-admin role
cat audit.log | jq -r '
  select(.objectRef.resource | test("rolebinding")) |
  select(.verb == "create") |
  select(.requestObject.roleRef.name == "cluster-admin") |
  [.stageTimestamp, .user.username, .objectRef.namespace // "cluster",
   .objectRef.name, (.requestObject.subjects[0].name // "unknown")] |
  @tsv
'

Pod Exec Investigation

Copy# All exec sessions (command in URI params)
cat audit.log | jq -r '
  select(.objectRef.subresource == "exec") |
  [.stageTimestamp, .user.username, .objectRef.namespace,
   .objectRef.name, .requestURI] |
  @tsv
'

# Exec by humans (non-service-accounts) — should be rare in prod
cat audit.log | jq -r '
  select(.objectRef.subresource == "exec") |
  select(.user.username | startswith("system:serviceaccount") | not) |
  [.stageTimestamp, .user.username, .objectRef.namespace,
   .objectRef.name, .requestURI] |
  @tsv
'

Unauthorized Access Patterns

Copy# All 403 Forbidden responses — RBAC denials
cat audit.log | jq -r '
  select(.responseStatus.code == 403) |
  [.stageTimestamp, .user.username, .verb,
   .objectRef.resource, .objectRef.namespace, .objectRef.name] |
  @tsv
'

# Top users by 403 count (credential probing detection)
cat audit.log | jq -r '
  select(.responseStatus.code == 403) |
  .user.username
' | sort | uniq -c | sort -rn | head -10

# 401 Unauthorized (authentication failures)
cat audit.log | jq -r '
  select(.responseStatus.code == 401) |
  [.stageTimestamp, .user.username, .sourceIPs[0], .requestURI] |
  @tsv
'

Node and Kubelet Activity

Copy# Nodes accessing other nodes' resources (NodeRestriction violation)
cat audit.log | jq -r '
  select(.user.groups[] == "system:nodes") |
  select(.user.username != ("system:node:" + .objectRef.name)) |
  select(.objectRef.resource == "nodes") |
  [.stageTimestamp, .user.username, .verb, .objectRef.name] |
  @tsv
'

Dynamic Audit Webhooks

Dynamic audit webhooks (auditregistration.k8s.io) allow configuring audit sinks via Kubernetes API objects without restarting the API server. This feature is alpha/beta in older versions — check availability in your cluster version.

Falco as an Audit Sink

Falco can receive Kubernetes audit events via webhook and apply its rule engine to detect security-relevant patterns in real time.

Copy# Configure audit webhook to send to Falco's audit endpoint
# webhook-audit-config.yaml
clusters:
- name: falco
  cluster:
    server: http://falco.falco-system.svc.cluster.local:8765/k8s-audit
contexts:
- name: falco
  context:
    cluster: falco
    user: ""
current-context: falco

Copy# Falco rule triggered by audit event: detect anonymous kubectl access
- rule: K8s Anonymous Request
  desc: Detect requests from system:anonymous or system:unauthenticated
  condition: >
    ka.user.name in ("system:anonymous", "system:unauthenticated")
  output: >
    Anonymous request to API server
    (user=%ka.user.name verb=%ka.verb uri=%ka.uri
     response=%ka.response.code sourceip=%ka.source.ip)
  priority: CRITICAL
  source: k8s_audit

- rule: K8s ClusterAdmin Binding Created
  desc: Detect creation of ClusterRoleBindings granting cluster-admin
  condition: >
    ka.verb = "create" and
    ka.target.resource = "clusterrolebindings" and
    ka.req.binding.role = "cluster-admin"
  output: >
    cluster-admin ClusterRoleBinding created
    (user=%ka.user.name binding=%ka.target.name subject=%ka.req.binding.subjects)
  priority: CRITICAL
  source: k8s_audit

- rule: K8s Secret Access
  desc: Detect reads of Kubernetes Secrets
  condition: >
    ka.target.resource = "secrets" and
    ka.verb in ("get", "list") and
    not ka.user.name startswith "system:serviceaccount:kube-system:"
  output: >
    Kubernetes Secret accessed
    (user=%ka.user.name secret=%ka.target.name ns=%ka.target.namespace)
  priority: WARNING
  source: k8s_audit

Compliance Requirements

Audit Log Integrity

Copy# Audit logs must be protected from modification
# Ship to immutable append-only storage immediately:

# 1. S3 Object Lock (compliance mode)
aws s3api put-object-lock-configuration \
  --bucket k8s-audit-logs \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Years":1}}}'

# 2. Restrict who can access audit logs via RBAC
# Audit logs should NOT be readable by workload service accounts

# 3. WORM storage at OS level (if keeping on-disk)
# chattr +a /var/log/kubernetes/audit/audit.log  (append-only)

Metrics, Alerts & Runbooks

Key Metrics

Alerts

Copy# Alert: Audit events being dropped
- alert: AuditEventsDropped
  expr: increase(apiserver_audit_requests_rejected_total[5m]) > 0
  for: 1m
  severity: critical
  annotations:
    summary: "Audit events being dropped — compliance gap"

# Alert: High rate of 403 responses (potential attack probe)
- alert: HighForbiddenRate
  expr: >
    sum(rate(apiserver_request_total{code="403"}[5m])) > 10
  for: 5m
  annotations:
    summary: "High rate of 403 responses — possible RBAC probing"

# Alert: cluster-admin binding created
- alert: ClusterAdminBindingCreated
  expr: increase(falco_events_total{rule="K8s ClusterAdmin Binding Created"}[5m]) > 0
  for: 0m
  severity: critical
  annotations:
    summary: "cluster-admin ClusterRoleBinding created — immediate investigation required"

# Alert: Audit log file not growing (logging broken)
- alert: AuditLogStale
  expr: rate(apiserver_audit_event_total[5m]) == 0
  for: 10m
  annotations:
    summary: "No audit events generated in 10 minutes — audit logging may be broken"

Runbooks

Best Practices