Gateway API

The next-generation Kubernetes ingress API — expressive, extensible, and role-oriented. Gateway API replaces Ingress for advanced traffic management while preserving a clean separation between infrastructure operators and application developers.

What This Page Covers
  • Gateway API resource model: GatewayClass, Gateway, HTTPRoute, GRPCRoute, TCPRoute, UDPRoute, TLSRoute, ReferenceGrant
  • Role-based design: infrastructure owner vs application developer separation
  • HTTPRoute: matches (host/path/header/query/method), filters, backendRefs, weights
  • Traffic splitting and canary with HTTPRoute weights
  • Header-based routing, URL rewriting, request/response header modification
  • RequestMirror filter for shadow traffic
  • ReferenceGrant: cross-namespace routing security model
  • BackendTLSPolicy: mTLS to backends, certificate validation
  • GRPCRoute: method/service matching, gRPC-specific semantics
  • TCPRoute and UDPRoute for L4 load balancing
  • TLSRoute: passthrough and terminate modes
  • ParentReference: multi-gateway listener binding
  • Controller implementations: Cilium, Contour, NGINX Gateway Fabric, Istio, Envoy Gateway, Traefik
  • Status conditions on Gateway and Route resources
  • Migration from Ingress to Gateway API patterns
  • Conformance profiles and implementation tiers (Core/Extended/Implementation-specific)
  • 7 metrics + 4 alerting rules + 5 troubleshooting runbooks
  • Best practices: shared gateways, namespace isolation, policy attachment

Why Gateway API Exists

Ingress (covered in 05-ingress.html) was never designed for the routing complexity modern platforms require. It lacks native support for traffic splitting, header-based routing, protocol awareness, and cross-namespace delegation. Every controller vendor ended up encoding these capabilities as proprietary annotations — creating a fragmented ecosystem where NGINX annotations don't work on Traefik, and vice versa.

Gateway API is a SIG-Network project that became GA in Kubernetes 1.31. It solves these problems by:

  • Expressiveness — traffic splitting, header routing, request mirroring, URL rewriting are first-class API concepts, not annotations.
  • Extensibility — a well-defined extension points model via policy attachment (BackendTLSPolicy, etc.).
  • Role separation — GatewayClass/Gateway owned by infra team; HTTPRoute owned by app team. Cross-namespace safely controlled by ReferenceGrant.
  • Portability — Core conformance is standardized; moving between controllers requires minimal manifest changes.
ℹ️
API Version Status

GatewayClass, Gateway, and HTTPRoute are v1 (GA) since gateway.networking.k8s.io/v1. GRPCRoute is v1 (GA) since 1.31. TCPRoute, UDPRoute, TLSRoute, and BackendTLSPolicy remain in v1alpha2. ReferenceGrant is v1beta1.

Resource Model

Gateway API introduces a hierarchy of resources, each scoped to a different concern and owned by a different persona:

Cluster Scope Namespace Scope ━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━ GatewayClass HTTPRoute ─────────┐ (infra team defines) GRPCRoute ├── backendRef → Service (same ns) controller: cilium.io/gateway TCPRoute │ or cross-ns + ReferenceGrant UDPRoute │ Namespace: gateway-infra TLSRoute │ Gateway ReferenceGrant ─────┘ parentRef → GatewayClass BackendTLSPolicy listeners: - name: https port: 443 protocol: HTTPS tls: certificateRefs: [Secret] allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: "true"

GatewayClass

GatewayClass is cluster-scoped. It identifies which controller implementation handles Gateways that reference this class — analogous to IngressClass but richer:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller
  description: "Cilium Gateway API implementation"
  parametersRef:                    # optional: controller-specific config CRD
    group: cilium.io
    kind: CiliumGatewayConfig
    name: default
    namespace: kube-system

Gateway

A Gateway represents a deployment of a load-balancing infrastructure (e.g., a provisioned cloud LB or an Envoy Deployment). It is typically owned by the infrastructure team and lives in a shared namespace:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: gateway-infra
spec:
  gatewayClassName: cilium
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"    # only namespaces with this label
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate                  # Terminate | Passthrough
        certificateRefs:
          - name: prod-tls-secret
            kind: Secret
      allowedRoutes:
        namespaces:
          from: All                      # or: Same | Selector
    - name: grpc-tls
      port: 8443
      protocol: HTTPS                    # HTTPS for gRPC-TLS
      tls:
        mode: Terminate
        certificateRefs:
          - name: grpc-tls-secret

After provisioning, the Gateway controller writes status conditions:

kubectl get gateway prod-gateway -n gateway-infra -o yaml
# status:
#   addresses:
#     - type: IPAddress
#       value: 203.0.113.42
#   conditions:
#     - type: Accepted
#       status: "True"
#     - type: Programmed
#       status: "True"

Role Separation

This is Gateway API's most important design principle. Three distinct personas each own different resources:

Infrastructure Owner

  • Creates GatewayClass (cluster-scoped)
  • Deploys and manages Gateway objects
  • Controls which namespaces can attach routes via allowedRoutes
  • Manages TLS certificates on listeners
  • Owns: kube-system, gateway-infra namespaces

Application Developer

  • Creates HTTPRoute/GRPCRoute in their own namespace
  • References a Gateway via parentRef
  • Defines routing rules, traffic splitting, header manipulation
  • Cannot modify Gateway or GatewayClass
  • For cross-namespace backend refs, needs ReferenceGrant approval

Cluster Operator (optional 3rd persona)

  • Attaches policy resources (BackendTLSPolicy, etc.) to infra
  • Manages cross-namespace ReferenceGrant approvals
  • Owns cluster-wide security posture for routing

Namespace Label Required

  • For from: Selector, the application namespace needs the label
  • kubectl label namespace my-app gateway-access=true
  • Without it, HTTPRoute attachment is rejected (status shows NotAllowedByListeners)

HTTPRoute — Full Reference

HTTPRoute is the workhorse of Gateway API. It expresses HTTP/HTTPS routing rules with first-class support for path matching, header matching, traffic splitting, and request/response transformation.

Basic Routing

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: store-route
  namespace: my-app                  # app developer's namespace
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra       # must match Gateway namespace
      sectionName: https             # bind to specific listener by name
  hostnames:
    - "store.example.com"
    - "*.example.com"                # wildcard hostnames supported
  rules:
    - matches:
        - path:
            type: PathPrefix         # Exact | PathPrefix | RegularExpression
            value: /api
          headers:
            - name: X-Version
              type: Exact            # Exact | RegularExpression
              value: "v2"
      backendRefs:
        - name: api-v2-svc
          port: 8080
          weight: 100
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend-svc
          port: 80

Traffic Splitting (Canary / Blue-Green)

Unlike Ingress canary which requires two Ingress objects, Gateway API handles it in a single HTTPRoute with weight fields on backendRefs:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-route
  namespace: my-app
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
  hostnames: ["app.example.com"]
  rules:
    - backendRefs:
        - name: app-stable
          port: 80
          weight: 90        # 90% traffic to stable
        - name: app-canary
          port: 80
          weight: 10        # 10% traffic to canary
ℹ️
Weight Semantics

Weights are proportional — they don't need to sum to 100. weight: 90, weight: 10 is equivalent to weight: 9, weight: 1. A weight of 0 means no traffic is sent to that backend (useful to drain a backend).

Header-Based Routing

rules:
  # Route internal users to canary based on header
  - matches:
      - headers:
          - name: X-Canary-User
            type: Exact
            value: "true"
    backendRefs:
      - name: app-canary
        port: 80

  # Route by query parameter
  - matches:
      - queryParams:
          - name: debug
            type: Exact
            value: "1"
    backendRefs:
      - name: app-debug
        port: 80

  # Method-based routing
  - matches:
      - method: POST
        path:
          type: PathPrefix
          value: /api/write
    backendRefs:
      - name: write-service
        port: 8080

HTTPRoute Filters

Filters transform requests and responses inline. Multiple filters can be applied per rule or per backendRef:

Filter TypeScopeDescription
RequestHeaderModifierrule / backendAdd, set, or remove request headers before forwarding
ResponseHeaderModifierrule / backendAdd, set, or remove response headers
RequestRedirectruleReturn 301/302/307/308 redirect (scheme, hostname, path, statusCode)
URLRewriterule / backendRewrite hostname and/or path before forwarding (ReplacePrefixMatch)
RequestMirrorruleShadow copy of request to a second backend (fire-and-forget)
ExtensionRefrule / backendController-specific extension (e.g., rate limiting policy)
rules:
  - matches:
      - path:
          type: PathPrefix
          value: /api
    filters:
      # Strip /api prefix before forwarding
      - type: URLRewrite
        urlRewrite:
          path:
            type: ReplacePrefixMatch
            replacePrefixMatch: /      # /api/users → /users

      # Add tracing header
      - type: RequestHeaderModifier
        requestHeaderModifier:
          add:
            - name: X-Forwarded-Prefix
              value: /api
          remove:
            - X-Internal-Debug

      # Shadow 10% to mirror service (fire-and-forget, response ignored)
      - type: RequestMirror
        requestMirror:
          backendRef:
            name: shadow-service
            port: 80

    backendRefs:
      - name: api-service
        port: 8080

  # HTTPS redirect on HTTP listener
  - matches:
      - path:
          type: PathPrefix
          value: /
    filters:
      - type: RequestRedirect
        requestRedirect:
          scheme: https
          statusCode: 301

ReferenceGrant — Cross-Namespace Security

By default, HTTPRoute can only reference backends (backendRef) in the same namespace. Cross-namespace references require explicit approval from the target namespace via a ReferenceGrant object. This prevents a compromised application namespace from routing traffic to services in other namespaces.

# In the TARGET namespace (where the Service lives)
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-store-to-payments
  namespace: payments             # the namespace being accessed
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: store            # the namespace making the reference
  to:
    - group: ""
      kind: Service
      name: payment-svc           # optional: restrict to specific service name
                                  # omit to allow any Service in this namespace

The same mechanism protects Gateway TLS certificates. A Gateway referencing a Secret in another namespace also needs a ReferenceGrant from that Secret's namespace.

⚠️
ReferenceGrant is in the destination namespace

The from field specifies who is allowed to reference, and the grant lives in the namespace being referenced. The namespace owner grants access — never the requester. This ensures namespace owners retain control over their resources.

GRPCRoute

GRPCRoute (GA in v1.31) provides gRPC-aware routing with service and method matching. Unlike HTTPRoute with gRPC, it understands gRPC framing natively:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-route
  namespace: my-app
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
      sectionName: grpc-tls
  hostnames: ["grpc.example.com"]
  rules:
    - matches:
        - method:
            type: Exact                # Exact | RegularExpression
            service: com.example.UserService
            method: GetUser            # omit to match all methods in service
      backendRefs:
        - name: user-service-grpc
          port: 9090
          weight: 80
        - name: user-service-grpc-v2
          port: 9090
          weight: 20
    - matches:
        - method:
            service: com.example.AdminService  # match all AdminService RPCs
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Admin-Route
                value: "true"
      backendRefs:
        - name: admin-service-grpc
          port: 9090

L4 Routes: TCPRoute, UDPRoute, TLSRoute

These resources are still in v1alpha2 but are widely used for non-HTTP load balancing:

TCPRoute

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: postgres-route
  namespace: database
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
      sectionName: postgres         # listener: port 5432, protocol TCP
  rules:
    - backendRefs:
        - name: postgres-primary
          port: 5432
          weight: 100

TLSRoute (Passthrough)

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: db-tls-passthrough
  namespace: database
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
      sectionName: db-tls           # listener: port 5433, protocol TLS, mode Passthrough
  hostnames:
    - "db.internal.example.com"
  rules:
    - backendRefs:
        - name: postgres-tls
          port: 5432
ℹ️
TLSRoute vs HTTPS listener

TLSRoute is for passthrough mode — the Gateway forwards encrypted TLS without decrypting it (SNI-based routing). The backend handles TLS termination. Use this for databases, MQTT brokers, or any service requiring end-to-end encryption. For standard HTTPS termination at the gateway, use an HTTPRoute with an HTTPS listener.

BackendTLSPolicy — mTLS to Backends

BackendTLSPolicy (v1alpha3) configures TLS from the Gateway to the backend service — independent of whether the client connection is TLS or plain HTTP. This enables mTLS or one-way TLS for all traffic within the cluster:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: api-backend-tls
  namespace: my-app
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: api-service             # applies to this Service's port 8443
      namespace: my-app
  validation:
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: backend-ca-cert       # CA that signed the backend's certificate
    hostname: api.internal          # SNI hostname for cert verification
    wellKnownCACertificates: System  # alternative: use system CA bundle

BackendTLSPolicy follows the direct policy attachment model — it attaches directly to a Service, not to a route. This means any route (HTTPRoute, GRPCRoute) that references that Service will automatically use TLS when BackendTLSPolicy is present.

ParentReference — Multi-Listener Binding

An HTTPRoute can bind to multiple Gateways or multiple listeners on the same Gateway. Each parentRef entry can specify an optional sectionName (listener name) and port:

spec:
  parentRefs:
    # Bind to HTTPS listener on prod gateway
    - name: prod-gateway
      namespace: gateway-infra
      sectionName: https
      port: 443

    # Also bind to staging gateway (different namespace)
    - name: staging-gateway
      namespace: staging-infra
      sectionName: https
      port: 443

Route status is per-parentRef — each attachment reports its own Accepted and ResolvedRefs conditions:

kubectl get httproute store-route -n my-app -o jsonpath='{.status.parents}'
# [
#   {
#     "parentRef": {"name":"prod-gateway","namespace":"gateway-infra","sectionName":"https"},
#     "conditions": [
#       {"type":"Accepted","status":"True","reason":"Accepted"},
#       {"type":"ResolvedRefs","status":"True","reason":"ResolvedRefs"}
#     ]
#   }
# ]

Status Conditions Reference

Gateway Conditions

ConditionTrue MeaningFalse / Unknown Causes
AcceptedGatewayClass exists, controller is runningGatewayClass not found; no controller
ProgrammedGateway is ready to route (LB provisioned, listeners up)LB provisioning failed; cert missing
ReadyDeprecated alias for Programmed (v0.8+)Use Programmed instead

Listener Conditions

ConditionTrue MeaningCommon False Reasons
AcceptedListener config is validUnsupported protocol; conflicting port
ProgrammedListener is accepting connectionsTLS cert missing; port conflict
ResolvedRefsAll certificate refs resolvedSecret not found; ReferenceGrant missing
ConflictedNo conflict with other listenersDuplicate hostname+port combination

HTTPRoute Conditions

ConditionTrue MeaningCommon False Reasons
AcceptedRoute is attached to the gateway listenerNotAllowedByListeners; NoMatchingParent; NoMatchingListenerHostname
ResolvedRefsAll backendRefs foundBackendNotFound; InvalidKind; RefNotPermitted (needs ReferenceGrant)
PartiallyInvalidSome rules invalid but others workConflicting match rules within the route

Controller Implementations

Production Ready

Cilium Gateway API

  • Controller: io.cilium/gateway-controller
  • Implements HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute
  • Integrated with Cilium Network Policies
  • No Envoy dependency required (uses eBPF for L4)
  • Install: --set gatewayAPI.enabled=true
Production Ready

Envoy Gateway

  • Controller: gateway.envoyproxy.io/gatewayclass-controller
  • CNCF project; Envoy as data plane
  • Full Core + Extended conformance
  • HTTPRoute, GRPCRoute, TCPRoute, UDPRoute, TLSRoute
  • BackendTLSPolicy, ClientTrafficPolicy, BackendTrafficPolicy
Production Ready

NGINX Gateway Fabric

  • Controller: gateway.nginx.org/nginx-gateway-controller
  • nginxinc official Gateway API impl (not kubernetes/ingress-nginx)
  • Core conformance; HTTPRoute, GRPCRoute
  • NginxProxy CRD for nginx.conf tuning
Production Ready

Istio

  • Controller: istio.io/gateway-controller
  • Uses existing Envoy-based infrastructure
  • Gateway API as first-class alternative to VirtualService/Gateway
  • Recommended for new Istio installations
Production Ready

Contour

  • Controller: projectcontour.io/gateway-controller
  • Envoy data plane; strong TLS support
  • HTTPRoute, GRPCRoute, TLSRoute, TCPRoute
  • ContourDeployment CRD for provisioner config
Production Ready

Traefik

  • Controller: traefik.io/gateway-controller
  • Gateway API alongside IngressRoute CRDs
  • HTTPRoute Core + most Extended features
  • Middleware attachment via ExtensionRef

Conformance Profiles

Gateway API defines three tiers of conformance. Check an implementation's conformance report before choosing:

TierFeatures RequiredNotes
CoreBasic HTTPRoute path/header matching, TLS termination, basic filtersAll conformant implementations must pass these tests
ExtendedTraffic splitting, URLRewrite, RequestMirror, timeout, retry, RegExp matchingOptional but increasingly expected for production use
Implementation-SpecificExtensionRef filters, controller-specific CRDsDocumented via individual implementation; not portable

Installation

Gateway API CRDs are not bundled with Kubernetes — install them separately. Two channels available:

# Standard channel (stable: GatewayClass, Gateway, HTTPRoute, GRPCRoute, ReferenceGrant)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

# Experimental channel (adds TCPRoute, UDPRoute, TLSRoute, BackendTLSPolicy, etc.)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml

# Verify CRDs installed
kubectl get crd | grep gateway.networking.k8s.io

Then install your chosen controller. Example with Envoy Gateway:

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.1.0 \
  --namespace envoy-gateway-system \
  --create-namespace

# Verify
kubectl get pods -n envoy-gateway-system
kubectl get gatewayclass

Migration from Ingress

Most NGINX Ingress configurations map directly to HTTPRoute equivalents. Key differences:

Ingress PatternGateway API EquivalentNotes
Ingress host + path rulesHTTPRoute hostnames + rules.matchesDirect equivalent; path type mapping
nginx.ingress.kubernetes.io/rewrite-targetHTTPRoute URLRewrite filterNo annotation; first-class filter
Canary Ingress (canary-weight: 10)HTTPRoute backendRef weight: 10Single route instead of two objects
nginx.ingress.kubernetes.io/ssl-redirect: trueHTTPRoute RequestRedirect filter on HTTP listenerExplicit scheme redirect
IngressClass default annotationGatewayClass used by default GatewayMore explicit; no annotation needed
nginx.ingress.kubernetes.io/limit-rpsExtensionRef to controller-specific RateLimit policyNot yet standardized in Core
cert-manager Ingress annotationcert-manager Gateway + HTTPRoute annotationcert-manager v1.14+ supports Gateway API
auth-url annotationExtensionRef to auth filter (controller-specific)Envoy Gateway has EnvoyPatchPolicy

cert-manager with Gateway API

# cert-manager v1.14+ supports Gateway API natively
# Annotation on Gateway resource:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: gateway-infra
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod   # same as Ingress
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls     # cert-manager creates this Secret

Advanced Patterns

Shared Gateway Multi-Tenancy

A single Gateway can serve hundreds of application teams. Each team creates HTTPRoutes in their namespace pointing at the shared Gateway. The Gateway's allowedRoutes controls which namespaces can attach:

# Infrastructure team creates one Gateway per environment
# Gateway in: gateway-infra namespace
# App teams create HTTPRoutes in their own namespaces

# Team A: namespace "team-a" (has label gateway-access: "true")
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: team-a-app
  namespace: team-a
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
  hostnames: ["app-a.company.com"]
  rules:
    - backendRefs:
        - name: app-a-svc
          port: 80

# Team B: namespace "team-b"
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: team-b-app
  namespace: team-b
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-infra
  hostnames: ["app-b.company.com"]
  rules:
    - backendRefs:
        - name: app-b-svc
          port: 80

Timeouts and Retries (Extended)

rules:
  - matches:
      - path:
          type: PathPrefix
          value: /api
    timeouts:
      request: 30s              # total request timeout (client → backend)
      backendRequest: 10s       # per-attempt timeout to backend
    retry:                      # Extended conformance
      codes: [503, 504]         # retry on these status codes
      attempts: 3
      backoff: 250ms
    backendRefs:
      - name: api-service
        port: 8080

Session Persistence (Extended)

rules:
  - backendRefs:
      - name: session-app
        port: 80
    sessionPersistence:
      sessionName: "shop-session"
      absoluteTimeout: 1h
      idleTimeout: 30m
      type: Cookie              # Cookie | Header
      cookieConfig:
        lifetimeType: Session   # Session | Permanent

Ingress vs Gateway API Comparison

For a more complete Ingress reference, see 05-ingress.html. This table focuses on the architectural tradeoffs:

DimensionIngressGateway API
API stabilitynetworking.k8s.io/v1 (GA)gateway.networking.k8s.io/v1 (GA for HTTPRoute)
Traffic splittingTwo Ingress objects + annotationsNative: single HTTPRoute with weights
Header routingAnnotation (implementation-specific)Native: HTTPRoute matches.headers
URL rewritingnginx: rewrite-target annotationNative: URLRewrite filter
Role separationNone (all in one Ingress object)GatewayClass/Gateway vs HTTPRoute
Cross-namespaceNot supported nativelyReferenceGrant
L4 routingNot supportedTCPRoute, UDPRoute, TLSRoute
gRPCImplementation-specific (usually works)Native: GRPCRoute
TLS to backendsAnnotation-basedBackendTLSPolicy
PortabilityLow (annotations are implementation-specific)High (Core conformance is standardized)
ComplexityLow for basic use casesHigher initial setup; less operational complexity over time
Ecosystem maturityExtremely mature (5+ years)Mature; all major controllers support it (2024+)
When to choose Gateway API
  • You need traffic splitting, header routing, or request mirroring natively
  • Multiple teams share a cluster and need strong routing isolation
  • You need L4 (TCP/UDP) routing alongside HTTP
  • You're starting a new cluster — Gateway API is the forward path
ℹ️
When to keep Ingress
  • Existing clusters with stable Ingress configurations and no plans to migrate
  • Simple HTTP routing without splitting or header matching
  • Using a controller that doesn't yet have full Gateway API conformance

Metrics, Alerting & Troubleshooting

Key Metrics to Watch

MetricSourceWhat It Tells You
gateway_api_gateway_programmed_countControllerNumber of Gateways in Programmed state; drop = infra problem
gateway_api_httproute_accepted_countControllerRoutes successfully attached; watch for drops after deployment
envoy_cluster_upstream_rq_totalEnvoy / Cilium proxyTotal requests to each backend cluster
envoy_cluster_upstream_rq_time_bucketEnvoy proxyBackend response latency histogram
envoy_http_downstream_rq_5xxEnvoy proxy5xx rate at the gateway — spikes indicate backend failures
envoy_http_downstream_cx_activeEnvoy proxyActive connections; high values may indicate connection leak
nginx_ingress_controller_requestsNGINX GFRequest rate per host/status (if using NGINX Gateway Fabric)

Alerting Rules

# Alert: HTTPRoute not accepted
- alert: HTTPRouteNotAccepted
  expr: |
    kube_httproute_status_conditions{condition="Accepted",status="False"} > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "HTTPRoute {{ $labels.name }} in {{ $labels.namespace }} is not accepted"

# Alert: Gateway not programmed
- alert: GatewayNotProgrammed
  expr: |
    kube_gateway_status_conditions{condition="Programmed",status="False"} > 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Gateway {{ $labels.name }} is not programmed"

# Alert: High error rate through gateway
- alert: GatewayHighErrorRate
  expr: |
    rate(envoy_http_downstream_rq_5xx[5m]) /
    rate(envoy_http_downstream_rq_total[5m]) > 0.05
  for: 5m
  labels:
    severity: warning

# Alert: Backend response latency P99 > 2s
- alert: GatewayBackendHighLatency
  expr: |
    histogram_quantile(0.99,
      rate(envoy_cluster_upstream_rq_time_bucket[5m])
    ) > 2000
  for: 10m
  labels:
    severity: warning

Troubleshooting Runbooks

Runbook 1: HTTPRoute Not Accepted

# Check route status
kubectl get httproute my-route -n my-app -o yaml | grep -A20 status

# Common reasons and fixes:
# NotAllowedByListeners → namespace missing gateway-access label
kubectl label namespace my-app gateway-access=true

# NoMatchingListenerHostname → route hostname doesn't match listener hostname
# NoMatchingParent → Gateway name/namespace/sectionName is wrong
# RefNotPermitted → ReferenceGrant missing for cross-namespace backendRef

Runbook 2: Gateway Not Programmed / No Address

# Check gateway status
kubectl get gateway prod-gateway -n gateway-infra -o yaml

# Check controller pod logs
kubectl logs -n envoy-gateway-system -l app=gateway-helm -f

# Common causes:
# - GatewayClass controller not running
# - TLS certificate Secret missing or in wrong namespace + no ReferenceGrant
# - Cloud LB provisioning failure (check controller events)
kubectl describe gateway prod-gateway -n gateway-infra

Runbook 3: ResolvedRefs False (backendRef not found)

# Check route status for ResolvedRefs condition
kubectl get httproute my-route -n my-app -o jsonpath='{.status.parents[*].conditions}'

# BackendNotFound: Service doesn't exist or wrong name/port
kubectl get svc api-service -n my-app

# RefNotPermitted: cross-namespace backendRef without ReferenceGrant
# Create ReferenceGrant in the target namespace:
cat <

Runbook 4: Traffic Splitting Not Working (all traffic to one backend)

# Verify HTTPRoute has correct weights
kubectl get httproute canary-route -n my-app -o jsonpath='{.spec.rules[*].backendRefs}'

# Check both services have ready endpoints
kubectl get endpointslices -n my-app

# Check if controller supports traffic splitting (Extended conformance)
kubectl get gatewayclass cilium -o yaml | grep -i conformance

# Generate test traffic and measure split
for i in $(seq 1 100); do curl -s https://app.example.com/ -o /dev/null -w "%{http_code}\n"; done

Runbook 5: 502 Bad Gateway (backend connection refused)

# 502 usually means gateway can't reach backend
# Check pod readiness
kubectl get pods -n my-app -l app=api-service

# Check if Service has endpoints
kubectl get endpoints api-service -n my-app

# If BackendTLSPolicy is configured: verify backend is serving TLS on the right port
kubectl exec -n my-app deploy/api-service -- openssl s_server -help 2>&1 | head

# Check if port matches between Service.spec.ports and BackendTLSPolicy targetRef
kubectl describe backendtlspolicy api-backend-tls -n my-app

Best Practices

  1. Use shared Gateways, not per-team Gateways — one Gateway per environment tier (prod/staging) reduces cloud LB cost and simplifies certificate management. Use allowedRoutes.namespaces for isolation.
  2. Always specify sectionName in parentRef — binding to a specific listener by name prevents surprise attachments when new listeners are added.
  3. Put ReferenceGrants in the namespace being accessed — as a namespace owner, review and approve ReferenceGrant objects to maintain security boundaries.
  4. Use BackendTLSPolicy for internal mTLS — don't rely on unencrypted pod-to-pod traffic for sensitive services.
  5. Set explicit hostnames on HTTPRoutes — avoiding wildcard hostnames (*) prevents accidental traffic capture from other routes.
  6. Monitor status conditions in CI/CD — after deploying an HTTPRoute, verify Accepted: True and ResolvedRefs: True before marking deployment successful.
  7. Use traffic splitting for zero-downtime deploys — shift from weight:0/100 → 10/90 → 50/50 → 100/0 instead of hard cutover.
  8. Install experimental CRDs only if you need them — TCPRoute/UDPRoute are alpha; standard-install.yaml is safer for production unless you explicitly need L4 routing.