Gateway API
The next-generation Kubernetes ingress API — expressive, extensible, and role-oriented. Gateway API replaces Ingress for advanced traffic management while preserving a clean separation between infrastructure operators and application developers.
What This Page Covers
- Gateway API resource model: GatewayClass, Gateway, HTTPRoute, GRPCRoute, TCPRoute, UDPRoute, TLSRoute, ReferenceGrant
- Role-based design: infrastructure owner vs application developer separation
- HTTPRoute: matches (host/path/header/query/method), filters, backendRefs, weights
- Traffic splitting and canary with HTTPRoute weights
- Header-based routing, URL rewriting, request/response header modification
- RequestMirror filter for shadow traffic
- ReferenceGrant: cross-namespace routing security model
- BackendTLSPolicy: mTLS to backends, certificate validation
- GRPCRoute: method/service matching, gRPC-specific semantics
- TCPRoute and UDPRoute for L4 load balancing
- TLSRoute: passthrough and terminate modes
- ParentReference: multi-gateway listener binding
- Controller implementations: Cilium, Contour, NGINX Gateway Fabric, Istio, Envoy Gateway, Traefik
- Status conditions on Gateway and Route resources
- Migration from Ingress to Gateway API patterns
- Conformance profiles and implementation tiers (Core/Extended/Implementation-specific)
- 7 metrics + 4 alerting rules + 5 troubleshooting runbooks
- Best practices: shared gateways, namespace isolation, policy attachment
Why Gateway API Exists
Ingress (covered in 05-ingress.html) was never designed for the routing complexity modern platforms require. It lacks native support for traffic splitting, header-based routing, protocol awareness, and cross-namespace delegation. Every controller vendor ended up encoding these capabilities as proprietary annotations — creating a fragmented ecosystem where NGINX annotations don't work on Traefik, and vice versa.
Gateway API is a SIG-Network project that became GA in Kubernetes 1.31. It solves these problems by:
- Expressiveness — traffic splitting, header routing, request mirroring, URL rewriting are first-class API concepts, not annotations.
- Extensibility — a well-defined extension points model via policy attachment (BackendTLSPolicy, etc.).
- Role separation — GatewayClass/Gateway owned by infra team; HTTPRoute owned by app team. Cross-namespace safely controlled by ReferenceGrant.
- Portability — Core conformance is standardized; moving between controllers requires minimal manifest changes.
GatewayClass, Gateway, and HTTPRoute are v1 (GA) since gateway.networking.k8s.io/v1. GRPCRoute is v1 (GA) since 1.31. TCPRoute, UDPRoute, TLSRoute, and BackendTLSPolicy remain in v1alpha2. ReferenceGrant is v1beta1.
Resource Model
Gateway API introduces a hierarchy of resources, each scoped to a different concern and owned by a different persona:
GatewayClass
GatewayClass is cluster-scoped. It identifies which controller implementation handles Gateways that reference this class — analogous to IngressClass but richer:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: cilium
spec:
controllerName: io.cilium/gateway-controller
description: "Cilium Gateway API implementation"
parametersRef: # optional: controller-specific config CRD
group: cilium.io
kind: CiliumGatewayConfig
name: default
namespace: kube-system
Gateway
A Gateway represents a deployment of a load-balancing infrastructure (e.g., a provisioned cloud LB or an Envoy Deployment). It is typically owned by the infrastructure team and lives in a shared namespace:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: prod-gateway
namespace: gateway-infra
spec:
gatewayClassName: cilium
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
gateway-access: "true" # only namespaces with this label
- name: https
port: 443
protocol: HTTPS
tls:
mode: Terminate # Terminate | Passthrough
certificateRefs:
- name: prod-tls-secret
kind: Secret
allowedRoutes:
namespaces:
from: All # or: Same | Selector
- name: grpc-tls
port: 8443
protocol: HTTPS # HTTPS for gRPC-TLS
tls:
mode: Terminate
certificateRefs:
- name: grpc-tls-secret
After provisioning, the Gateway controller writes status conditions:
kubectl get gateway prod-gateway -n gateway-infra -o yaml
# status:
# addresses:
# - type: IPAddress
# value: 203.0.113.42
# conditions:
# - type: Accepted
# status: "True"
# - type: Programmed
# status: "True"
Role Separation
This is Gateway API's most important design principle. Three distinct personas each own different resources:
Infrastructure Owner
- Creates GatewayClass (cluster-scoped)
- Deploys and manages Gateway objects
- Controls which namespaces can attach routes via
allowedRoutes - Manages TLS certificates on listeners
- Owns:
kube-system,gateway-infranamespaces
Application Developer
- Creates HTTPRoute/GRPCRoute in their own namespace
- References a Gateway via
parentRef - Defines routing rules, traffic splitting, header manipulation
- Cannot modify Gateway or GatewayClass
- For cross-namespace backend refs, needs ReferenceGrant approval
Cluster Operator (optional 3rd persona)
- Attaches policy resources (BackendTLSPolicy, etc.) to infra
- Manages cross-namespace ReferenceGrant approvals
- Owns cluster-wide security posture for routing
Namespace Label Required
- For
from: Selector, the application namespace needs the label kubectl label namespace my-app gateway-access=true- Without it, HTTPRoute attachment is rejected (status shows
NotAllowedByListeners)
HTTPRoute — Full Reference
HTTPRoute is the workhorse of Gateway API. It expresses HTTP/HTTPS routing rules with first-class support for path matching, header matching, traffic splitting, and request/response transformation.
Basic Routing
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: store-route
namespace: my-app # app developer's namespace
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra # must match Gateway namespace
sectionName: https # bind to specific listener by name
hostnames:
- "store.example.com"
- "*.example.com" # wildcard hostnames supported
rules:
- matches:
- path:
type: PathPrefix # Exact | PathPrefix | RegularExpression
value: /api
headers:
- name: X-Version
type: Exact # Exact | RegularExpression
value: "v2"
backendRefs:
- name: api-v2-svc
port: 8080
weight: 100
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: frontend-svc
port: 80
Traffic Splitting (Canary / Blue-Green)
Unlike Ingress canary which requires two Ingress objects, Gateway API handles it in a single HTTPRoute with weight fields on backendRefs:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: canary-route
namespace: my-app
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
hostnames: ["app.example.com"]
rules:
- backendRefs:
- name: app-stable
port: 80
weight: 90 # 90% traffic to stable
- name: app-canary
port: 80
weight: 10 # 10% traffic to canary
Weights are proportional — they don't need to sum to 100. weight: 90, weight: 10 is equivalent to weight: 9, weight: 1. A weight of 0 means no traffic is sent to that backend (useful to drain a backend).
Header-Based Routing
rules:
# Route internal users to canary based on header
- matches:
- headers:
- name: X-Canary-User
type: Exact
value: "true"
backendRefs:
- name: app-canary
port: 80
# Route by query parameter
- matches:
- queryParams:
- name: debug
type: Exact
value: "1"
backendRefs:
- name: app-debug
port: 80
# Method-based routing
- matches:
- method: POST
path:
type: PathPrefix
value: /api/write
backendRefs:
- name: write-service
port: 8080
HTTPRoute Filters
Filters transform requests and responses inline. Multiple filters can be applied per rule or per backendRef:
| Filter Type | Scope | Description |
|---|---|---|
RequestHeaderModifier | rule / backend | Add, set, or remove request headers before forwarding |
ResponseHeaderModifier | rule / backend | Add, set, or remove response headers |
RequestRedirect | rule | Return 301/302/307/308 redirect (scheme, hostname, path, statusCode) |
URLRewrite | rule / backend | Rewrite hostname and/or path before forwarding (ReplacePrefixMatch) |
RequestMirror | rule | Shadow copy of request to a second backend (fire-and-forget) |
ExtensionRef | rule / backend | Controller-specific extension (e.g., rate limiting policy) |
rules:
- matches:
- path:
type: PathPrefix
value: /api
filters:
# Strip /api prefix before forwarding
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: / # /api/users → /users
# Add tracing header
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: X-Forwarded-Prefix
value: /api
remove:
- X-Internal-Debug
# Shadow 10% to mirror service (fire-and-forget, response ignored)
- type: RequestMirror
requestMirror:
backendRef:
name: shadow-service
port: 80
backendRefs:
- name: api-service
port: 8080
# HTTPS redirect on HTTP listener
- matches:
- path:
type: PathPrefix
value: /
filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
ReferenceGrant — Cross-Namespace Security
By default, HTTPRoute can only reference backends (backendRef) in the same namespace. Cross-namespace references require explicit approval from the target namespace via a ReferenceGrant object. This prevents a compromised application namespace from routing traffic to services in other namespaces.
# In the TARGET namespace (where the Service lives)
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: allow-store-to-payments
namespace: payments # the namespace being accessed
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: store # the namespace making the reference
to:
- group: ""
kind: Service
name: payment-svc # optional: restrict to specific service name
# omit to allow any Service in this namespace
The same mechanism protects Gateway TLS certificates. A Gateway referencing a Secret in another namespace also needs a ReferenceGrant from that Secret's namespace.
The from field specifies who is allowed to reference, and the grant lives in the namespace being referenced. The namespace owner grants access — never the requester. This ensures namespace owners retain control over their resources.
GRPCRoute
GRPCRoute (GA in v1.31) provides gRPC-aware routing with service and method matching. Unlike HTTPRoute with gRPC, it understands gRPC framing natively:
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
name: grpc-route
namespace: my-app
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
sectionName: grpc-tls
hostnames: ["grpc.example.com"]
rules:
- matches:
- method:
type: Exact # Exact | RegularExpression
service: com.example.UserService
method: GetUser # omit to match all methods in service
backendRefs:
- name: user-service-grpc
port: 9090
weight: 80
- name: user-service-grpc-v2
port: 9090
weight: 20
- matches:
- method:
service: com.example.AdminService # match all AdminService RPCs
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: X-Admin-Route
value: "true"
backendRefs:
- name: admin-service-grpc
port: 9090
L4 Routes: TCPRoute, UDPRoute, TLSRoute
These resources are still in v1alpha2 but are widely used for non-HTTP load balancing:
TCPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
name: postgres-route
namespace: database
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
sectionName: postgres # listener: port 5432, protocol TCP
rules:
- backendRefs:
- name: postgres-primary
port: 5432
weight: 100
TLSRoute (Passthrough)
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: db-tls-passthrough
namespace: database
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
sectionName: db-tls # listener: port 5433, protocol TLS, mode Passthrough
hostnames:
- "db.internal.example.com"
rules:
- backendRefs:
- name: postgres-tls
port: 5432
TLSRoute is for passthrough mode — the Gateway forwards encrypted TLS without decrypting it (SNI-based routing). The backend handles TLS termination. Use this for databases, MQTT brokers, or any service requiring end-to-end encryption. For standard HTTPS termination at the gateway, use an HTTPRoute with an HTTPS listener.
BackendTLSPolicy — mTLS to Backends
BackendTLSPolicy (v1alpha3) configures TLS from the Gateway to the backend service — independent of whether the client connection is TLS or plain HTTP. This enables mTLS or one-way TLS for all traffic within the cluster:
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
name: api-backend-tls
namespace: my-app
spec:
targetRefs:
- group: ""
kind: Service
name: api-service # applies to this Service's port 8443
namespace: my-app
validation:
caCertificateRefs:
- group: ""
kind: ConfigMap
name: backend-ca-cert # CA that signed the backend's certificate
hostname: api.internal # SNI hostname for cert verification
wellKnownCACertificates: System # alternative: use system CA bundle
BackendTLSPolicy follows the direct policy attachment model — it attaches directly to a Service, not to a route. This means any route (HTTPRoute, GRPCRoute) that references that Service will automatically use TLS when BackendTLSPolicy is present.
ParentReference — Multi-Listener Binding
An HTTPRoute can bind to multiple Gateways or multiple listeners on the same Gateway. Each parentRef entry can specify an optional sectionName (listener name) and port:
spec:
parentRefs:
# Bind to HTTPS listener on prod gateway
- name: prod-gateway
namespace: gateway-infra
sectionName: https
port: 443
# Also bind to staging gateway (different namespace)
- name: staging-gateway
namespace: staging-infra
sectionName: https
port: 443
Route status is per-parentRef — each attachment reports its own Accepted and ResolvedRefs conditions:
kubectl get httproute store-route -n my-app -o jsonpath='{.status.parents}'
# [
# {
# "parentRef": {"name":"prod-gateway","namespace":"gateway-infra","sectionName":"https"},
# "conditions": [
# {"type":"Accepted","status":"True","reason":"Accepted"},
# {"type":"ResolvedRefs","status":"True","reason":"ResolvedRefs"}
# ]
# }
# ]
Status Conditions Reference
Gateway Conditions
| Condition | True Meaning | False / Unknown Causes |
|---|---|---|
Accepted | GatewayClass exists, controller is running | GatewayClass not found; no controller |
Programmed | Gateway is ready to route (LB provisioned, listeners up) | LB provisioning failed; cert missing |
Ready | Deprecated alias for Programmed (v0.8+) | Use Programmed instead |
Listener Conditions
| Condition | True Meaning | Common False Reasons |
|---|---|---|
Accepted | Listener config is valid | Unsupported protocol; conflicting port |
Programmed | Listener is accepting connections | TLS cert missing; port conflict |
ResolvedRefs | All certificate refs resolved | Secret not found; ReferenceGrant missing |
Conflicted | No conflict with other listeners | Duplicate hostname+port combination |
HTTPRoute Conditions
| Condition | True Meaning | Common False Reasons |
|---|---|---|
Accepted | Route is attached to the gateway listener | NotAllowedByListeners; NoMatchingParent; NoMatchingListenerHostname |
ResolvedRefs | All backendRefs found | BackendNotFound; InvalidKind; RefNotPermitted (needs ReferenceGrant) |
PartiallyInvalid | Some rules invalid but others work | Conflicting match rules within the route |
Controller Implementations
Cilium Gateway API
- Controller:
io.cilium/gateway-controller - Implements HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute
- Integrated with Cilium Network Policies
- No Envoy dependency required (uses eBPF for L4)
- Install:
--set gatewayAPI.enabled=true
Envoy Gateway
- Controller:
gateway.envoyproxy.io/gatewayclass-controller - CNCF project; Envoy as data plane
- Full Core + Extended conformance
- HTTPRoute, GRPCRoute, TCPRoute, UDPRoute, TLSRoute
- BackendTLSPolicy, ClientTrafficPolicy, BackendTrafficPolicy
NGINX Gateway Fabric
- Controller:
gateway.nginx.org/nginx-gateway-controller - nginxinc official Gateway API impl (not kubernetes/ingress-nginx)
- Core conformance; HTTPRoute, GRPCRoute
- NginxProxy CRD for nginx.conf tuning
Istio
- Controller:
istio.io/gateway-controller - Uses existing Envoy-based infrastructure
- Gateway API as first-class alternative to VirtualService/Gateway
- Recommended for new Istio installations
Contour
- Controller:
projectcontour.io/gateway-controller - Envoy data plane; strong TLS support
- HTTPRoute, GRPCRoute, TLSRoute, TCPRoute
- ContourDeployment CRD for provisioner config
Traefik
- Controller:
traefik.io/gateway-controller - Gateway API alongside IngressRoute CRDs
- HTTPRoute Core + most Extended features
- Middleware attachment via ExtensionRef
Conformance Profiles
Gateway API defines three tiers of conformance. Check an implementation's conformance report before choosing:
| Tier | Features Required | Notes |
|---|---|---|
| Core | Basic HTTPRoute path/header matching, TLS termination, basic filters | All conformant implementations must pass these tests |
| Extended | Traffic splitting, URLRewrite, RequestMirror, timeout, retry, RegExp matching | Optional but increasingly expected for production use |
| Implementation-Specific | ExtensionRef filters, controller-specific CRDs | Documented via individual implementation; not portable |
Installation
Gateway API CRDs are not bundled with Kubernetes — install them separately. Two channels available:
# Standard channel (stable: GatewayClass, Gateway, HTTPRoute, GRPCRoute, ReferenceGrant)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml
# Experimental channel (adds TCPRoute, UDPRoute, TLSRoute, BackendTLSPolicy, etc.)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml
# Verify CRDs installed
kubectl get crd | grep gateway.networking.k8s.io
Then install your chosen controller. Example with Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm \
--version v1.1.0 \
--namespace envoy-gateway-system \
--create-namespace
# Verify
kubectl get pods -n envoy-gateway-system
kubectl get gatewayclass
Migration from Ingress
Most NGINX Ingress configurations map directly to HTTPRoute equivalents. Key differences:
| Ingress Pattern | Gateway API Equivalent | Notes |
|---|---|---|
Ingress host + path rules | HTTPRoute hostnames + rules.matches | Direct equivalent; path type mapping |
nginx.ingress.kubernetes.io/rewrite-target | HTTPRoute URLRewrite filter | No annotation; first-class filter |
Canary Ingress (canary-weight: 10) | HTTPRoute backendRef weight: 10 | Single route instead of two objects |
nginx.ingress.kubernetes.io/ssl-redirect: true | HTTPRoute RequestRedirect filter on HTTP listener | Explicit scheme redirect |
IngressClass default annotation | GatewayClass used by default Gateway | More explicit; no annotation needed |
nginx.ingress.kubernetes.io/limit-rps | ExtensionRef to controller-specific RateLimit policy | Not yet standardized in Core |
cert-manager Ingress annotation | cert-manager Gateway + HTTPRoute annotation | cert-manager v1.14+ supports Gateway API |
auth-url annotation | ExtensionRef to auth filter (controller-specific) | Envoy Gateway has EnvoyPatchPolicy |
cert-manager with Gateway API
# cert-manager v1.14+ supports Gateway API natively
# Annotation on Gateway resource:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: prod-gateway
namespace: gateway-infra
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod # same as Ingress
spec:
gatewayClassName: cilium
listeners:
- name: https
port: 443
protocol: HTTPS
hostname: "*.example.com"
tls:
mode: Terminate
certificateRefs:
- name: wildcard-tls # cert-manager creates this Secret
Advanced Patterns
Shared Gateway Multi-Tenancy
A single Gateway can serve hundreds of application teams. Each team creates HTTPRoutes in their namespace pointing at the shared Gateway. The Gateway's allowedRoutes controls which namespaces can attach:
# Infrastructure team creates one Gateway per environment
# Gateway in: gateway-infra namespace
# App teams create HTTPRoutes in their own namespaces
# Team A: namespace "team-a" (has label gateway-access: "true")
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: team-a-app
namespace: team-a
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
hostnames: ["app-a.company.com"]
rules:
- backendRefs:
- name: app-a-svc
port: 80
# Team B: namespace "team-b"
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: team-b-app
namespace: team-b
spec:
parentRefs:
- name: prod-gateway
namespace: gateway-infra
hostnames: ["app-b.company.com"]
rules:
- backendRefs:
- name: app-b-svc
port: 80
Timeouts and Retries (Extended)
rules:
- matches:
- path:
type: PathPrefix
value: /api
timeouts:
request: 30s # total request timeout (client → backend)
backendRequest: 10s # per-attempt timeout to backend
retry: # Extended conformance
codes: [503, 504] # retry on these status codes
attempts: 3
backoff: 250ms
backendRefs:
- name: api-service
port: 8080
Session Persistence (Extended)
rules:
- backendRefs:
- name: session-app
port: 80
sessionPersistence:
sessionName: "shop-session"
absoluteTimeout: 1h
idleTimeout: 30m
type: Cookie # Cookie | Header
cookieConfig:
lifetimeType: Session # Session | Permanent
Ingress vs Gateway API Comparison
For a more complete Ingress reference, see 05-ingress.html. This table focuses on the architectural tradeoffs:
| Dimension | Ingress | Gateway API |
|---|---|---|
| API stability | networking.k8s.io/v1 (GA) | gateway.networking.k8s.io/v1 (GA for HTTPRoute) |
| Traffic splitting | Two Ingress objects + annotations | Native: single HTTPRoute with weights |
| Header routing | Annotation (implementation-specific) | Native: HTTPRoute matches.headers |
| URL rewriting | nginx: rewrite-target annotation | Native: URLRewrite filter |
| Role separation | None (all in one Ingress object) | GatewayClass/Gateway vs HTTPRoute |
| Cross-namespace | Not supported natively | ReferenceGrant |
| L4 routing | Not supported | TCPRoute, UDPRoute, TLSRoute |
| gRPC | Implementation-specific (usually works) | Native: GRPCRoute |
| TLS to backends | Annotation-based | BackendTLSPolicy |
| Portability | Low (annotations are implementation-specific) | High (Core conformance is standardized) |
| Complexity | Low for basic use cases | Higher initial setup; less operational complexity over time |
| Ecosystem maturity | Extremely mature (5+ years) | Mature; all major controllers support it (2024+) |
- You need traffic splitting, header routing, or request mirroring natively
- Multiple teams share a cluster and need strong routing isolation
- You need L4 (TCP/UDP) routing alongside HTTP
- You're starting a new cluster — Gateway API is the forward path
- Existing clusters with stable Ingress configurations and no plans to migrate
- Simple HTTP routing without splitting or header matching
- Using a controller that doesn't yet have full Gateway API conformance
Metrics, Alerting & Troubleshooting
Key Metrics to Watch
| Metric | Source | What It Tells You |
|---|---|---|
gateway_api_gateway_programmed_count | Controller | Number of Gateways in Programmed state; drop = infra problem |
gateway_api_httproute_accepted_count | Controller | Routes successfully attached; watch for drops after deployment |
envoy_cluster_upstream_rq_total | Envoy / Cilium proxy | Total requests to each backend cluster |
envoy_cluster_upstream_rq_time_bucket | Envoy proxy | Backend response latency histogram |
envoy_http_downstream_rq_5xx | Envoy proxy | 5xx rate at the gateway — spikes indicate backend failures |
envoy_http_downstream_cx_active | Envoy proxy | Active connections; high values may indicate connection leak |
nginx_ingress_controller_requests | NGINX GF | Request rate per host/status (if using NGINX Gateway Fabric) |
Alerting Rules
# Alert: HTTPRoute not accepted
- alert: HTTPRouteNotAccepted
expr: |
kube_httproute_status_conditions{condition="Accepted",status="False"} > 0
for: 5m
labels:
severity: warning
annotations:
summary: "HTTPRoute {{ $labels.name }} in {{ $labels.namespace }} is not accepted"
# Alert: Gateway not programmed
- alert: GatewayNotProgrammed
expr: |
kube_gateway_status_conditions{condition="Programmed",status="False"} > 0
for: 10m
labels:
severity: critical
annotations:
summary: "Gateway {{ $labels.name }} is not programmed"
# Alert: High error rate through gateway
- alert: GatewayHighErrorRate
expr: |
rate(envoy_http_downstream_rq_5xx[5m]) /
rate(envoy_http_downstream_rq_total[5m]) > 0.05
for: 5m
labels:
severity: warning
# Alert: Backend response latency P99 > 2s
- alert: GatewayBackendHighLatency
expr: |
histogram_quantile(0.99,
rate(envoy_cluster_upstream_rq_time_bucket[5m])
) > 2000
for: 10m
labels:
severity: warning
Troubleshooting Runbooks
Runbook 1: HTTPRoute Not Accepted
# Check route status
kubectl get httproute my-route -n my-app -o yaml | grep -A20 status
# Common reasons and fixes:
# NotAllowedByListeners → namespace missing gateway-access label
kubectl label namespace my-app gateway-access=true
# NoMatchingListenerHostname → route hostname doesn't match listener hostname
# NoMatchingParent → Gateway name/namespace/sectionName is wrong
# RefNotPermitted → ReferenceGrant missing for cross-namespace backendRef
Runbook 2: Gateway Not Programmed / No Address
# Check gateway status
kubectl get gateway prod-gateway -n gateway-infra -o yaml
# Check controller pod logs
kubectl logs -n envoy-gateway-system -l app=gateway-helm -f
# Common causes:
# - GatewayClass controller not running
# - TLS certificate Secret missing or in wrong namespace + no ReferenceGrant
# - Cloud LB provisioning failure (check controller events)
kubectl describe gateway prod-gateway -n gateway-infra
Runbook 3: ResolvedRefs False (backendRef not found)
# Check route status for ResolvedRefs condition
kubectl get httproute my-route -n my-app -o jsonpath='{.status.parents[*].conditions}'
# BackendNotFound: Service doesn't exist or wrong name/port
kubectl get svc api-service -n my-app
# RefNotPermitted: cross-namespace backendRef without ReferenceGrant
# Create ReferenceGrant in the target namespace:
cat <
Runbook 4: Traffic Splitting Not Working (all traffic to one backend)
# Verify HTTPRoute has correct weights
kubectl get httproute canary-route -n my-app -o jsonpath='{.spec.rules[*].backendRefs}'
# Check both services have ready endpoints
kubectl get endpointslices -n my-app
# Check if controller supports traffic splitting (Extended conformance)
kubectl get gatewayclass cilium -o yaml | grep -i conformance
# Generate test traffic and measure split
for i in $(seq 1 100); do curl -s https://app.example.com/ -o /dev/null -w "%{http_code}\n"; done
Runbook 5: 502 Bad Gateway (backend connection refused)
# 502 usually means gateway can't reach backend
# Check pod readiness
kubectl get pods -n my-app -l app=api-service
# Check if Service has endpoints
kubectl get endpoints api-service -n my-app
# If BackendTLSPolicy is configured: verify backend is serving TLS on the right port
kubectl exec -n my-app deploy/api-service -- openssl s_server -help 2>&1 | head
# Check if port matches between Service.spec.ports and BackendTLSPolicy targetRef
kubectl describe backendtlspolicy api-backend-tls -n my-app
Best Practices
- Use shared Gateways, not per-team Gateways — one Gateway per environment tier (prod/staging) reduces cloud LB cost and simplifies certificate management. Use
allowedRoutes.namespacesfor isolation. - Always specify
sectionNamein parentRef — binding to a specific listener by name prevents surprise attachments when new listeners are added. - Put ReferenceGrants in the namespace being accessed — as a namespace owner, review and approve ReferenceGrant objects to maintain security boundaries.
- Use BackendTLSPolicy for internal mTLS — don't rely on unencrypted pod-to-pod traffic for sensitive services.
- Set explicit hostnames on HTTPRoutes — avoiding wildcard hostnames (
*) prevents accidental traffic capture from other routes. - Monitor status conditions in CI/CD — after deploying an HTTPRoute, verify
Accepted: TrueandResolvedRefs: Truebefore marking deployment successful. - Use traffic splitting for zero-downtime deploys — shift from weight:0/100 → 10/90 → 50/50 → 100/0 instead of hard cutover.
- Install experimental CRDs only if you need them — TCPRoute/UDPRoute are alpha; standard-install.yaml is safer for production unless you explicitly need L4 routing.