Kubernetes Events

The fourth observability signal: event structure, retention, kubectl usage, event exporters, kubernetes-event-exporter, custom event recording, alerting on events, and production event pipeline design.

What Are Kubernetes Events

Kubernetes Events are first-class API objects (core/v1 Event) written to the API server by controllers, the kubelet, the scheduler, and other system components. They describe what happened to a Kubernetes object — a pod was scheduled, an image pull failed, a container was OOMKilled, a deployment scaled up.

Events are the "fourth observability signal" alongside metrics, logs, and traces. They are unique because they are generated automatically by the cluster itself with no application code changes. Every significant state transition in the cluster produces an event.

Events vs Audit Logs vs Application Logs

Event Object Structure

kubectl get event -n production -o yaml | head -80Copy

apiVersion: v1
kind: Event
metadata:
  name: order-service-7d5f9-xk2pq.17c4a3f0b8e2d1a5   # pod-name.hex-suffix
  namespace: production
  creationTimestamp: "2024-01-15T10:23:45Z"

# The resource this event is about
involvedObject:
  apiVersion: v1
  kind: Pod
  name: order-service-7d5f9-xk2pq
  namespace: production
  uid: a1b2c3d4-e5f6-7890-abcd-ef1234567890
  resourceVersion: "98765"
  fieldPath: spec.containers{order-service}   # specific container, if applicable

# Classification
reason: OOMKilling             # machine-readable reason (CamelCase)
message: "Memory limit reached: Killing container with allowance 256Mi, used 256Mi"
type: Warning                  # Normal or Warning

# Deduplication and aggregation
count: 3                       # how many times this event has occurred
firstTimestamp: "2024-01-15T09:45:00Z"
lastTimestamp:  "2024-01-15T10:23:45Z"

# Who generated this event
source:
  component: kubelet
  host: ip-10-0-1-45.us-east-1.compute.internal

# v1.19+ fields (EventSeries for high-frequency events)
series:
  count: 3
  lastObservedTime: "2024-01-15T10:23:45.000000Z"

# v1.19+ reporting fields (replaces source)
reportingComponent: kubelet
reportingInstance: ip-10-0-1-45.us-east-1.compute.internal

# Related object (action target, if different from involvedObject)
related:
  apiVersion: v1
  kind: Node
  name: ip-10-0-1-45.us-east-1.compute.internal

action: Killing                # verb describing the action taken
eventTime: "2024-01-15T10:23:45.000000Z"   # MicroTime (v1.19+)Copy

Key Fields Reference

Event Deduplication

The Kubernetes event recorder deduplicates events that share the same involvedObject, reason, message, source, and type within a 10-minute window. Duplicate events increment count and update lastTimestamp rather than creating new Event objects. This prevents etcd flooding from hot loops but means you lose individual timestamps for high-frequency events.

Event Types and Sources

Event Types

Event Sources (Reporting Components)

kubectl events Usage

# List all events in a namespace (sorted by time, most recent last)
kubectl get events -n production

# Sort by most recent first
kubectl get events -n production --sort-by='.lastTimestamp'

# Only Warning events
kubectl get events -n production --field-selector type=Warning

# Events for a specific pod
kubectl get events -n production --field-selector involvedObject.name=order-service-7d5f9-xk2pq

# Events for a specific object kind
kubectl get events -n production --field-selector involvedObject.kind=Node

# Events by reason
kubectl get events -n production --field-selector reason=OOMKilling

# Cross-namespace (all namespaces)
kubectl get events -A --sort-by='.lastTimestamp'

# Only Warning events cluster-wide (most useful for incident triage)
kubectl get events -A --field-selector type=Warning --sort-by='.lastTimestamp'

# Watch events as they arrive
kubectl get events -n production -w

# Describe a specific resource to see associated events inline
kubectl describe pod order-service-7d5f9-xk2pq -n production
# Events section at the bottom shows events for this pod

# Custom columns (useful for quick triage)
kubectl get events -n production \
  -o custom-columns='TIME:.lastTimestamp,TYPE:.type,REASON:.reason,OBJECT:.involvedObject.name,MSG:.message' \
  --sort-by='.lastTimestamp' | tail -30

# JSON output for scripting
kubectl get events -n production -o json | \
  jq '.items[] | select(.type=="Warning") | {time:.lastTimestamp, reason:.reason, object:.involvedObject.name, msg:.message}'Copy

# kubectl events (1.26+)
kubectl events -n production
kubectl events -n production --for pod/order-service-7d5f9-xk2pq
kubectl events -n production --types=Warning
kubectl events -n production --for deployment/order-service --types=Warning
kubectl events -A --types=Warning --no-headers | sort -k1Copy

Important Event Reasons

The reason field is the most important field for programmatic filtering and alerting. These are the reasons that matter most for production operations:

Warning Events (Action Required)

Normal Events (Operational Visibility)

Retention and the TTL Problem

The 1-hour event TTL is the single most important operational fact about Kubernetes events. Most production incidents last longer than 1 hour, and retrospective analysis almost always requires events from before the incident started.

Changing the Default TTL

# /etc/kubernetes/manifests/kube-apiserver.yaml (static pod)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --event-ttl=4h        # extend to 4 hours
        # WARNING: longer TTL = more etcd storage consumed by events
        # Events are stored in etcd under /registry/events/
        # High-event clusters can produce GBs of event data per hourCopy

etcd Event Quota

The API server has a per-namespace event object limit (default 1000 events per namespace per object type). In a namespace with many pods crashing, events may be dropped before TTL expires. The --max-events-per-namespace flag on kube-apiserver (v1.21+) controls this:

- --max-events-per-namespace=1000   # default; increase if events are being droppedCopy

kubernetes-event-exporter

kubernetes-event-exporter (by Resmo, formerly opsgenie) is the most feature-rich and actively maintained event exporter. It watches the Events API and routes events to multiple backends based on configurable filters.

Architecture

Helm Install

helm repo add deliveryhero https://charts.deliveryhero.io
helm upgrade --install kubernetes-event-exporter deliveryhero/kubernetes-event-exporter \
  --namespace monitoring \
  --create-namespace \
  --set config.logLevel=warn \
  --set config.logFormat=json \
  --values event-exporter-values.yamlCopy

Production Configuration

# event-exporter-config.yaml (mounted as ConfigMap)
logLevel: warn
logFormat: json

# Cluster-wide label added to all exported events
clusterName: prod-us-east-1

receivers:
  # --- Loki: all events ---
  - name: loki
    loki:
      streamLabels:
        app: kubernetes-event-exporter
        cluster: prod-us-east-1
      url: http://loki-gateway.monitoring.svc/loki/api/v1/push
      headers:
        Content-Type: application/json
      layout:
        # Map event fields to Loki log labels and body
        namespace: "{{ .InvolvedObject.Namespace }}"
        name: "{{ .InvolvedObject.Name }}"
        kind: "{{ .InvolvedObject.Kind }}"
        reason: "{{ .Reason }}"
        type: "{{ .Type }}"
        message: "{{ .Message }}"
        component: "{{ .ReportingComponent }}"
        count: "{{ .Count }}"

  # --- Elasticsearch: all events ---
  - name: elasticsearch
    elasticsearch:
      hosts:
        - https://prod-logs-es-http.logging.svc:9200
      index: kubernetes-events
      indexFormat: kubernetes-events-{2006.01.02}   # daily index
      username: elastic
      password:
        valueFrom:
          secretKeyRef:
            name: elastic-credentials
            key: password
      tls:
        insecureSkipVerify: false
        caFile: /es-certs/ca.crt
      layout:
        timestamp: "{{ .LastTimestamp }}"
        namespace: "{{ .InvolvedObject.Namespace }}"
        name: "{{ .InvolvedObject.Name }}"
        kind: "{{ .InvolvedObject.Kind }}"
        reason: "{{ .Reason }}"
        message: "{{ .Message }}"
        type: "{{ .Type }}"
        count: "{{ .Count }}"
        component: "{{ .ReportingComponent }}"
        cluster: prod-us-east-1

  # --- Slack: Warning events only ---
  - name: slack-warnings
    slack:
      token:
        valueFrom:
          secretKeyRef:
            name: slack-token
            key: token
      channel: "#k8s-warnings"
      message: |
        *[{{ .Type }}]* `{{ .Reason }}` on `{{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}`
        *Namespace:* `{{ .InvolvedObject.Namespace }}`
        *Message:* {{ .Message }}
        *Count:* {{ .Count }} | *Last seen:* {{ .LastTimestamp }}

  # --- PagerDuty: critical Warning reasons ---
  - name: pagerduty-critical
    opsgenie:
      apiKey:
        valueFrom:
          secretKeyRef:
            name: opsgenie-key
            key: apiKey
      message: "K8s {{ .Reason }}: {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}"
      description: "{{ .Message }}"
      alias: "{{ .InvolvedObject.Namespace }}/{{ .InvolvedObject.Name }}/{{ .Reason }}"
      priority: P2

route:
  # Default: send all events to Loki and Elasticsearch
  routes:
    # Warning events → Loki + Elasticsearch + Slack
    - match:
        - type: Warning
      receivers:
        - loki
        - elasticsearch
        - slack-warnings

    # Critical Warning reasons → also page
    - match:
        - type: Warning
          reason:
            - OOMKilling
            - Evicted
            - BackoffLimitExceeded
            - NodeNotReady
      receivers:
        - pagerduty-critical

    # Normal events → Loki only (cheaper, not in Elasticsearch)
    - match:
        - type: Normal
      receivers:
        - loki

    # Exclude noisy Normal events from kube-system
    - drop:
        - type: Normal
          namespace: kube-system
          reason:
            - LeaderElection
            - Pulling
            - PulledCopy

RBAC for Event Exporter

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-event-exporter
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubernetes-event-exporter
rules:
  - apiGroups: [""]
    resources: [events, namespaces, nodes, pods]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, replicasets, statefulsets, daemonsets]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-event-exporter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-event-exporter
subjects:
  - kind: ServiceAccount
    name: kubernetes-event-exporter
    namespace: monitoringCopy

Deployment with Leader Election

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-event-exporter
  namespace: monitoring
spec:
  replicas: 2   # HA: leader election prevents duplicate events
  selector:
    matchLabels:
      app: kubernetes-event-exporter
  template:
    spec:
      serviceAccountName: kubernetes-event-exporter
      containers:
        - name: exporter
          image: ghcr.io/resmoio/kubernetes-event-exporter:v1.7
          args:
            - -conf=/data/config.yaml
            - -leader-election          # ensures only one replica exports at a time
            - -leader-election-id=event-exporter-lock
          resources:
            requests: {cpu: 50m, memory: 64Mi}
            limits: {cpu: 200m, memory: 256Mi}
          volumeMounts:
            - name: config
              mountPath: /data
      volumes:
        - name: config
          configMap:
            name: kubernetes-event-exporter-configCopy

kube-eventer

kube-eventer (Alibaba Cloud) is an alternative event exporter with native support for DingTalk, Kafka, Elasticsearch, InfluxDB, and more. Useful when kubernetes-event-exporter lacks a specific output plugin.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-eventer
  namespace: monitoring
spec:
  replicas: 1
  template:
    spec:
      serviceAccountName: kube-eventer
      containers:
        - name: kube-eventer
          image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.2.7-358d0d3-aliyun
          command:
            - /kube-eventer
            - --source=kubernetes:https://kubernetes.default
            # Elasticsearch sink
            - --sink=elasticsearch:https://prod-es:9200?sniff=false&index=kube-events&x-pack-enabled=true
            # DingTalk sink (optional)
            # - --sink=dingtalk:https://oapi.dingtalk.com/robot/send?level=warning&label=prod
          resources:
            requests: {cpu: 50m, memory: 64Mi}
            limits: {cpu: 200m, memory: 128Mi}Copy

Event Router Patterns

Fluent Bit as Event Collector

Rather than a dedicated event exporter, Fluent Bit can watch the Kubernetes Events API directly and route to the same Loki/Elasticsearch pipeline used for container logs:

[INPUT]
    Name              kubernetes_events
    # Connects to the K8s API and watches /api/v1/events
    Kube_URL          https://kubernetes.default.svc:443
    Kube_CA_File      /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File   /var/run/secrets/kubernetes.io/serviceaccount/token
    DB                /var/log/flb-events.db
    Interval_Sec      5

[FILTER]
    Name   grep
    Match  kube_events
    # Only Warning events
    Regex  type Warning

[OUTPUT]
    Name  loki
    Match kube_events
    Host  loki-gateway.monitoring.svc
    Labels job=k8s-events,cluster=prodCopy

OTel Collector k8sevents Receiver

receivers:
  k8s_events:
    auth_type: serviceAccount
    namespaces: []   # empty = all namespaces

processors:
  resource:
    attributes:
      - key: cluster
        value: prod-us-east-1
        action: insert
  filter/warnings_only:
    logs:
      log_record:
        - 'attributes["k8s.event.reason"] != "" and attributes["k8s.event.action"] != "Scheduled"'
  batch:
    timeout: 10s

exporters:
  loki:
    endpoint: http://loki-gateway.monitoring.svc/loki/api/v1/push
    labels:
      resource:
        cluster: ""
        k8s.namespace.name: ""

service:
  pipelines:
    logs/events:
      receivers: [k8s_events]
      processors: [resource, batch]
      exporters: [loki]Copy

Recording Custom Events

Kubernetes controllers and operators should emit events to report the outcome of reconciliation loops. This follows the standard Kubernetes operational model and makes operator behavior observable via kubectl describe and event exporters without any extra tooling.

Go: client-go EventRecorder

package controller

import (
    "context"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/client-go/tools/record"
    ctrl "sigs.k8s.io/controller-runtime"
)

type MyReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

func (r *MyReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // Create recorder with the component name (appears in event source)
    r.Recorder = mgr.GetEventRecorderFor("my-controller")
    return ctrl.NewControllerManagedBy(mgr).
        For(&myv1.MyResource{}).
        Complete(r)
}

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var resource myv1.MyResource
    if err := r.Get(ctx, req.NamespacedName, &resource); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Normal event: informational
    r.Recorder.Event(&resource, corev1.EventTypeNormal,
        "Reconciled",              // reason (CamelCase)
        "Resource reconciled successfully",  // message
    )

    // Warning event: something went wrong
    if err := r.doSomething(ctx, &resource); err != nil {
        r.Recorder.Eventf(&resource, corev1.EventTypeWarning,
            "SyncFailed",          // reason
            "Failed to sync resource: %v", err,  // formatted message
        )
        return ctrl.Result{}, err
    }

    // Annotation-based event (annotate the related object)
    r.Recorder.AnnotatedEventf(&resource, map[string]string{
        "reconcile-version": "v2",
    }, corev1.EventTypeNormal, "Updated", "Configuration updated to version %s", "v2")

    return ctrl.Result{}, nil
}Copy

Event Reason Naming Conventions

controller-runtime Event Recorder (operator-sdk style)

// In main.go / cmd/main.go (operator-sdk scaffold)
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme: scheme,
    // EventBroadcaster is configured automatically
    // EventRecorder is available via mgr.GetEventRecorderFor()
})

// Use in reconciler:
type DatabaseReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    db := &dbv1.Database{}
    if err := r.Get(ctx, req.NamespacedName, db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Record event on a *related* object (not the resource being reconciled)
    // e.g., record event on the Secret created by this controller
    secret := &corev1.Secret{}
    r.Recorder.Event(secret, corev1.EventTypeNormal, "Created",
        "Secret created for database credentials")

    return ctrl.Result{}, nil
}Copy

Alerting on Events

There are two approaches to alert on Kubernetes events: using kube-state-metrics (converts events to Prometheus metrics for PromQL alerting) and direct alerting from the event exporter pipeline (e.g., Loki alert rules or webhook routing).

kube-state-metrics: Event Metrics

kube-state-metrics (KSM) exposes event counts as Prometheus metrics, enabling event-based alerting via standard PrometheusRule without a separate event exporter:

# kube_event_count: count of events by type/reason/namespace/name
# Available in kube-state-metrics v2.8+
kube_event_count{type="Warning", reason="OOMKilling"}
kube_event_count{type="Warning", reason="BackOff"}
kube_event_count{type="Warning", reason="Evicted"}
kube_event_count{type="Warning", reason="FailedScheduling"}

# Rate of Warning events (increase = new events)
increase(kube_event_count{type="Warning"}[5m])

# Find namespaces with OOM kills in last 15 minutes
sum by (namespace) (increase(kube_event_count{type="Warning", reason="OOMKilling"}[15m]))Copy

PrometheusRule Alerts on Events

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-event-alerts
  namespace: monitoring
spec:
  groups:
    - name: kubernetes-events
      rules:
        - alert: KubernetesOOMKilling
          expr: |
            increase(kube_event_count{type="Warning", reason="OOMKilling"}[10m]) > 0
          labels:
            severity: warning
          annotations:
            summary: "OOMKill in {{ $labels.namespace }} ({{ $labels.name }})"
            description: "Container was OOMKilled. Increase memory limit or investigate memory leak."
            runbook_url: "https://wiki/runbooks/oomkilling"

        - alert: KubernetesCrashLoopBackOff
          expr: |
            increase(kube_event_count{type="Warning", reason="BackOff"}[5m]) > 3
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "CrashLoopBackOff in {{ $labels.namespace }} ({{ $labels.name }})"
            description: "Pod is in CrashLoopBackOff. Check container logs with --previous flag."

        - alert: KubernetesPodEvicted
          expr: |
            increase(kube_event_count{type="Warning", reason="Evicted"}[10m]) > 0
          labels:
            severity: warning
          annotations:
            summary: "Pod evicted in {{ $labels.namespace }}"
            description: "A pod was evicted due to node resource pressure."

        - alert: KubernetesFailedScheduling
          expr: |
            increase(kube_event_count{type="Warning", reason="FailedScheduling"}[5m]) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod cannot be scheduled in {{ $labels.namespace }}"
            description: "Scheduler cannot find a suitable node. Check resource requests and node capacity."

        - alert: KubernetesNodeNotReady
          expr: |
            increase(kube_event_count{type="Warning", reason="NodeNotReady"}[5m]) > 0
          labels:
            severity: critical
          annotations:
            summary: "Node entered NotReady state"
            description: "A node transitioned to NotReady. Pods may be evicted after tolerationSeconds."

        - alert: KubernetesImagePullFailure
          expr: |
            increase(kube_event_count{type="Warning", reason=~"Failed|ErrImagePull|ImagePullBackOff"}[5m]) > 0
          for: 3m
          labels:
            severity: warning
          annotations:
            summary: "Image pull failing in {{ $labels.namespace }}"
            description: "Container image cannot be pulled. Check imagePullSecret and registry reachability."Copy

Loki-Based Event Alerting

When events are forwarded to Loki via kubernetes-event-exporter or Fluent Bit, you can write LogQL alerts. This captures the full event message (not just counts) and allows richer filtering:

apiVersion: monitoring.coreos.com/v1alpha1
kind: PrometheusRule
metadata:
  name: loki-event-alerts
  namespace: monitoring
spec:
  groups:
    - name: loki-events
      interval: 1m
      rules:
        - alert: KubernetesWarningEventSpike
          expr: |
            sum by (namespace) (
              rate({job="k8s-events"} | json | type="Warning" [5m])
            ) > 5
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High rate of Warning events in {{ $labels.namespace }}"

        - alert: CriticalEventPattern
          expr: |
            count_over_time(
              {job="k8s-events"} | json | type="Warning"
                | reason =~ "OOMKilling|Evicted|BackoffLimitExceeded|NodeNotReady" [5m]
            ) > 0
          labels:
            severity: critical
          annotations:
            summary: "Critical Kubernetes event detected"Copy

Slack Alert Routing for Events

# kubernetes-event-exporter routing: critical events → PagerDuty, warnings → Slack
route:
  routes:
    - match:
        - type: Warning
          reason:
            - OOMKilling
            - NodeNotReady
            - Evicted
      throttle:
        # Prevent Slack flooding: max 10 messages per 5 minutes per reason
        period: 5m
        count: 10
      receivers:
        - slack-critical
        - elasticsearch

    - match:
        - type: Warning
      receivers:
        - slack-warnings
        - elasticsearchCopy

Metrics, Alerts & Runbooks

Key Metrics for the Event Pipeline

Operational Alert Rules

groups:
  - name: event-exporter-health
    rules:
      - alert: EventExporterSendErrors
        expr: rate(kubernetes_event_exporter_send_errors_total[5m]) > 0
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "kubernetes-event-exporter failing to send events — check receiver connectivity"

      - alert: EventExporterWatchErrors
        expr: increase(kubernetes_event_exporter_watch_errors_total[5m]) > 3
        labels: {severity: warning}
        annotations:
          summary: "Event exporter watch API errors — events may be missed"

      - alert: EventExporterDown
        expr: absent(kubernetes_event_exporter_events_total)
        for: 5m
        labels: {severity: critical}
        annotations:
          summary: "kubernetes-event-exporter is not running — no events are being exported"Copy

Runbooks

Querying Events in Loki

# All Warning events in last 1 hour
{job="k8s-events"} | json | type="Warning"

# OOMKilling events with details
{job="k8s-events"} | json | reason="OOMKilling"
  | line_format "{{.namespace}}/{{.name}}: {{.message}}"

# Events for a specific namespace
{job="k8s-events"} | json | namespace="payments"

# Count Warning events by reason (last 24h)
sum by (reason) (count_over_time({job="k8s-events"} | json | type="Warning" [24h]))

# Find all events for a specific pod
{job="k8s-events"} | json | name=~"order-service-.*"

# Events that correlate with a deployment (around rollout time)
{job="k8s-events"} | json | reason="ScalingReplicaSet" | name=~"order-service-.*"Copy

Best Practices

1. Deploy a persistent event exporter from day one. The 1-hour TTL makes events useless for any incident that lasts more than an hour. Run kubernetes-event-exporter (or equivalent) before you need it — you cannot retroactively recover expired events.
2. Send Warning events to an alerting channel. Not every Warning event needs a page, but they should appear in a low-urgency Slack channel where on-call engineers can spot patterns. Aggregate by reason to identify systemic issues.
3. Route critical event reasons to PagerDuty. OOMKilling, NodeNotReady, Evicted, and BackoffLimitExceeded should trigger alerting. Use throttling (max N per 5 minutes) to prevent alert fatigue from cascading failures.
4. Emit events from custom controllers. Every operator reconcile loop should emit a Normal event on success and a Warning event on failure. This makes controller behavior observable via kubectl describe with no additional tooling required.
5. Filter out LeaderElection and health-probe events. These fire every few seconds in kube-system and add noise without value. Exclude them in the event exporter routing rules before they reach Elasticsearch or Loki.
6. Do not store events in Elasticsearch unless you need full-text search. Loki is 10–20× cheaper per event record. Use Elasticsearch only if you need complex aggregation queries or full-text search on event messages. Forward to Loki for standard operational use.
7. Enable leader election in the event exporter. A single-replica event exporter is a single point of failure for your event pipeline. Run 2+ replicas with leader election to survive pod restarts without missing events.
8. Add cluster label to all exported events. In multi-cluster environments, always include a cluster label in exported events so you can distinguish events from different clusters when they land in a shared Loki/Elasticsearch instance.