Nginx in Kubernetes and debugging
Ingress controllers and the Gateway API
An Ingress controller is a reverse proxy that configures itself from Kubernetes Ingress objects. For years the most common one was the community ingress-nginx project, configured mostly through annotations such as nginx.ingress.kubernetes.io/proxy-body-size.
The Kubernetes project retired ingress-nginx in March 2026: no more releases, bug fixes or security patches. Clusters still running it should migrate. Note that F5's NGINX Ingress Controller (nginx/kubernetes-ingress) is a separate, still-maintained product with different annotations, so "we use Nginx ingress" needs a follow-up question.
The recommended direction is the Gateway API, the successor to Ingress:
| Ingress | Gateway API |
|---|---|
| One object mixes infrastructure and routing | GatewayClass (which implementation), Gateway (listeners, owned by the platform team), HTTPRoute (routes, owned by app teams) |
| Features via controller-specific annotations | Header matching, traffic weights, redirects and rewrites in the standard spec |
| HTTP only in the core spec | HTTP, gRPC, TLS and TCP routes |
Implementations include Envoy Gateway, Istio, Cilium, Kong, Traefik, NGINX Gateway Fabric and, on AWS, the AWS Load Balancer Controller and VPC Lattice. A migration plan: inventory every Ingress and its annotations, map each annotation to a Gateway API feature or a policy resource, run the new controller alongside the old one, move host names one at a time by DNS, then remove ingress-nginx.
Debugging with logs
The access log tells you what happened; the error log tells you why. Add timing fields to the access log:
log_format timed '$remote_addr "$request" $status $body_bytes_sent '
'rt=$request_time uct=$upstream_connect_time urt=$upstream_response_time '
'ua="$upstream_addr" us=$upstream_status';
$request_time is the total time; $upstream_response_time is the backend's part. A big gap between them points at a slow client or buffering, not the app.
| Symptom | Error log says | Usual cause |
|---|---|---|
| 502 Bad Gateway | connect() failed (111: Connection refused) while connecting to upstream |
App not listening, wrong port, container still starting |
| 502 | upstream prematurely closed connection |
App crashed mid-request, or its keep-alive timeout is shorter than Nginx's |
| 502 | no live upstreams |
Every server marked failed by max_fails |
| 504 Gateway Timeout | upstream timed out (110: Connection timed out) while reading response header |
App too slow for proxy_read_timeout, or a blocked connection |
| 413 | client intended to send too large body |
Raise client_max_body_size (and the Ingress annotation) |
| 499 (in access log) | — | The client gave up before Nginx answered (often a shorter timeout in front) |
Workflow: reproduce with curl -v -H 'Host: shop.example.com' http://127.0.0.1/path, read the matching error-log line, check nginx -T to see which server and location really apply, then test the upstream directly with curl from the proxy host.
Alternatives
| Proxy | Strength | Typical use |
|---|---|---|
| Nginx | Fast static serving, mature, simple config | Web servers, edge proxies, small ingress |
| HAProxy | Excellent L4/L7 balancing, active health checks, detailed stats | High-traffic load balancing, TCP services |
| Envoy | Dynamic config over APIs (xDS), rich observability, gRPC, retries | Service meshes, Gateway API implementations |
| Traefik | Auto-discovers routes from Docker and Kubernetes labels | Small clusters, Docker hosts |
| Caddy | Automatic HTTPS with almost no config | Small sites, internal tools |