Opscraft
Networking fundamentals · Lesson 8 of 9 · 5 min read

A troubleshooting method

Bottom-up, with evidence at each layer

  1. Define the failure precisely: who (one client, all clients?), what (DNS, connect, TLS, HTTP status?), since when, intermittent or constant, what changed.
  2. DNS — does the name resolve, to the expected IP, from the failing client? dig, getent hosts.
  3. Routing — is there a route? ip route get, VPC route tables, Reachability Analyzer.
  4. Filtering — SGs, NACLs, host firewalls, NetworkPolicies, proxies. Flow Logs show REJECTs.
  5. Transport — does the TCP handshake complete? nc -zv, tcpdump on both ends.
  6. TLS — openssl s_client -connect host:443 -servername host.
  7. Application — curl -v, status codes, app logs, LB access logs.
Enlarge diagramDefine itwho, what fails, sincewhen, what changedDNSdig, getent hostsRoutingip route get,route tablesFilteringSGs, NACLs, FlowLogsTransportnc -zv, tcpdumpTLSopenssl s_clientApplicationcurl -v, logs, LBlogs1234567
Figure 1. Troubleshoot bottom-up, proving each layer works before moving to the next.
Swipe sideways to see all of it, or tap to open it full screen.

Reading failure signatures

Symptom Most likely meaning
Connection refused Host reached; nothing listening on that port (RST) — service down, wrong port, bound to 127.0.0.1
Connection timed out Packets dropped — SG/NACL/firewall, missing route, wrong IP, host down
No route to host Routing failure or an ICMP reject from a firewall
Connection reset by peer Remote sent RST mid-connection — app crashed, idle timeout on a proxy/NAT, protocol mismatch
Name or service not known / NXDOMAIN DNS — wrong name, missing record, wrong resolver
Works by IP, not by hostname DNS (resolv.conf, resolver, the record, split-horizon)
Small requests work, large hang MTU / PMTUD
Intermittent timeouts only under load NAT port exhaustion, conntrack table full, LB limits, DNS throttling (VPC resolver per-ENI limits), ephemeral port exhaustion
TLS errors from one environment only Missing CA bundle, TLS-intercepting proxy, clock skew, SNI missing
Works from laptop, not server (or reverse) Different DNS, routes, egress firewall/SG, proxy settings, TLS trust store
Some users slow, others fine Geography/latency, a bad CDN edge or ISP path, DNS-based routing differences

Tools

dig +short api.example.com ; dig @10.0.0.2 api.internal          # DNS
ip -br a ; ip route ; ip route get 10.30.1.15                    # local config and routing
ss -tulnp                                                        # what's listening
nc -zv db.internal 5432                                          # TCP connect test
curl -v --connect-timeout 5 https://api.internal/health          # full HTTP + TLS
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null
traceroute -T -p 443 api.example.com ; mtr -rwc 50 api.example.com   # path, loss per hop
tcpdump -i any -nn host 10.30.1.15 and port 5432                  # packets
iperf3 -s / iperf3 -c host                                        # throughput between two hosts

Proving a firewall drop: run tcpdump on both ends — if SYNs leave the client but never arrive at the server, something in between drops them; if they arrive and SYN-ACKs leave but never reach the client, suspect the return path (NACL outbound rule, asymmetric routing).

Enlarge diagramClienttcpdump hereNetwork inbetweenSGs, NACLs, routesServertcpdump hereserver capture shows nothing: something in between drops itnever reaches the client: check outbound NACL rules and the returnrouteCase 1: lost on the way inSYN leaves the clientCase 2: lost on the way backSYN arrives at the serverSYN-ACK leaves the server
Figure 2. Proving where packets are dropped: capture on both ends at once.
Swipe sideways to see all of it, or tap to open it full screen.

Scenario walkthroughs

"Intermittent timeouts to an external API only at peak." Suspect NAT gateway port allocation errors (CloudWatch ErrorPortAllocation), conntrack limits on nodes, or DNS throttling. Fix: connection pooling/keep-alive, more NAT IPs, spreading traffic, NodeLocal DNSCache.

"Users far away see slow pages." Measure where time goes (curl -w timings). Fix with CloudFront for static and cacheable content, regional deployments with Route 53 latency routing, HTTP/2 or HTTP/3, fewer round trips.

"NAT gateway bill is huge." Use Flow Logs/Cost Explorer to find top talkers; add S3/DynamoDB gateway endpoints and ECR/Logs interface endpoints; avoid cross-AZ NAT traffic (one NAT per AZ, routes per AZ); cache images closer (pull-through cache).

"Packet loss suspected between two EC2 instances." mtr and ping -c 1000 for loss, iperf3 for throughput, check instance-type network limits (bandwidth, PPS — ethtool -S shows bw_in_allowance_exceeded, pps_allowance_exceeded on ENA), and placement.