Opscraft
The TCP handshake · Lesson 6 of 8 · 3 min read

Troubleshooting scenarios

"Connection timed out" for clients, server looks fine

Run tcpdump on the server for the port. No SYNs arriving → the problem is upstream: security group or NACL, route tables, the load balancer's target health or listener, or DNS pointing to the wrong IP. SYNs arriving but no SYN-ACK leaving → host firewall (iptables/nftables), accept queue full, or the service bound to another interface. SYN-ACKs leaving but clients still time out → return path (NACL outbound ephemeral ports, asymmetric routing through a firewall/NAT).

Clients see resets when pods terminate

During a rollout, pods receive SIGTERM while kube-proxy and the ALB are still sending them traffic; the app exits and the kernel RSTs connections. Fix with a preStop sleep, graceful shutdown that drains in-flight requests, and adequate terminationGracePeriodSeconds.

Enlarge diagramKubernetesPodALB / kube-proxyWithout preStopSIGTERMremove the pod from endpoints (takes a few seconds)exits at oncestill sends requestsRST: connection refused or resetWith a preStop sleep of ~10 spreStop: sleep 10, then SIGTERMremove the pod from endpointsin-flight requests finish
Figure 1. Why clients see resets during a rollout: the pod is told to stop while traffic is still being routed to it. A short preStop sleep lets routing catch up first.
Swipe sideways to see all of it, or tap to open it full screen.

Health checks pass, users fail

A TCP health check only proves the handshake completes — the kernel completes handshakes even if the application is deadlocked (as long as the accept queue isn't full). Use HTTP health checks against an endpoint that exercises the app.

Thousands of TIME_WAIT on a reverse proxy

The proxy opens a new upstream connection per request and closes it. Enable upstream keep-alive/connection pooling; consider tcp_tw_reuse.

NAT gateway connection failures under load

Too many simultaneous connections to the same destination IP:port exhaust the NAT's source ports (≈55,000 per NAT IP per destination). Reuse connections, add NAT IPs, or spread destinations.

"Cannot assign requested address"

Ephemeral port exhaustion on the client (see TIME_WAIT). Pool connections.

Means loss or congestion; TCP backs off, so latency rises and throughput falls. Check ss -ti retransmit counters, nstat TcpRetransSegs, the path with mtr, instance network allowances (ENA ethtool -S), and MTU.