Troubleshooting scenarios
"Connection timed out" for clients, server looks fine
Run tcpdump on the server for the port. No SYNs arriving → the problem is upstream: security group or NACL, route tables, the load balancer's target health or listener, or DNS pointing to the wrong IP. SYNs arriving but no SYN-ACK leaving → host firewall (iptables/nftables), accept queue full, or the service bound to another interface. SYN-ACKs leaving but clients still time out → return path (NACL outbound ephemeral ports, asymmetric routing through a firewall/NAT).
Clients see resets when pods terminate
During a rollout, pods receive SIGTERM while kube-proxy and the ALB are still sending them traffic; the app exits and the kernel RSTs connections. Fix with a preStop sleep, graceful shutdown that drains in-flight requests, and adequate terminationGracePeriodSeconds.
Health checks pass, users fail
A TCP health check only proves the handshake completes — the kernel completes handshakes even if the application is deadlocked (as long as the accept queue isn't full). Use HTTP health checks against an endpoint that exercises the app.
Thousands of TIME_WAIT on a reverse proxy
The proxy opens a new upstream connection per request and closes it. Enable upstream keep-alive/connection pooling; consider tcp_tw_reuse.
NAT gateway connection failures under load
Too many simultaneous connections to the same destination IP:port exhaust the NAT's source ports (≈55,000 per NAT IP per destination). Reuse connections, add NAT IPs, or spread destinations.
"Cannot assign requested address"
Ephemeral port exhaustion on the client (see TIME_WAIT). Pool connections.
Retransmissions are high on a link
Means loss or congestion; TCP backs off, so latency rises and throughput falls. Check ss -ti retransmit counters, nstat TcpRetransSegs, the path with mtr, instance network allowances (ENA ethtool -S), and MTU.