Opscraft
TCP vs UDP · Lesson 2 of 8 · 8 min read

TCP in depth

Properties

TCP (Transmission Control Protocol) provides:

  • Connection-oriented communication — a three-way handshake before data (detailed in the handshake topic) and an orderly close.
  • Reliable delivery — every byte is acknowledged; lost data is retransmitted.
  • In-order delivery — the receiver reassembles bytes in sequence.
  • Byte stream — no message boundaries; the application must frame messages (length prefixes, newlines, HTTP headers).
  • Flow control — the sender never overwhelms the receiver.
  • Congestion control — the sender adapts to network capacity.
  • Full duplex — both directions independently.

The unit is a segment.

The TCP header

Field Purpose
Source / destination port 16 bits each
Sequence number Byte position of this segment's first data byte
Acknowledgement number Next byte the sender of this segment expects to receive
Data offset Header length (header is 20 bytes minimum, up to 60 with options)
Flags SYN, ACK, FIN, RST, PSH, URG, ECE, CWR
Window size How many bytes the sender of this segment can receive (flow control)
Checksum Detects corruption
Urgent pointer Rarely used
Options MSS, window scale, SACK permitted, timestamps

Reliability: sequence numbers, ACKs and retransmission

Every byte has a sequence number. The receiver sends cumulative ACKs: "ACK 5001" means "I have everything up to byte 5000; send 5001 next".

Loss is detected two ways:

  1. Retransmission timeout (RTO) — no ACK within a timeout computed from measured round-trip time (minimum ~200 ms on Linux, doubling on each retry — exponential backoff). Timeouts are slow and severely cut throughput.
  2. Fast retransmit — the receiver keeps acknowledging the last in-order byte when later segments arrive; three duplicate ACKs tell the sender a segment is missing, and it retransmits immediately without waiting for the timeout.

SACK (selective acknowledgement) lets the receiver report exactly which blocks it has, so the sender retransmits only the gaps. Timestamps improve RTT measurement and protect against wrapped sequence numbers on fast links.

Enlarge diagramSenderReceiverbytes 1-1000ACK 1001bytes 1001-2000: lost on the waybytes 2001-3000ACK 1001 (duplicate 1)bytes 3001-4000ACK 1001 (duplicate 2)bytes 4001-5000ACK 1001 (duplicate 3)fast retransmit: bytes 1001-2000ACK 5001: everything received
Figure 1. How TCP recovers a lost segment quickly: the receiver keeps asking for the missing byte, and three duplicate ACKs trigger a fast retransmit.
Swipe sideways to see all of it, or tap to open it full screen.

Flow control: the receive window

Each side advertises a receive window (rwnd) — free space in its receive buffer. The sender may have at most that many unacknowledged bytes in flight. A slow application that doesn't read its socket shrinks the window to zero; the sender pauses ("zero window" in packet captures) and probes periodically.

The window field is only 16 bits (max 65,535 bytes); the window scaling option (negotiated in the SYNs) multiplies it, allowing windows of many megabytes — essential for fast, long-distance links.

Congestion control: why loss kills throughput

Flow control protects the receiver; congestion control protects the network. The sender keeps a congestion window (cwnd); it may send min(cwnd, rwnd) unacknowledged bytes.

  • Slow start — a new connection starts with a small cwnd (commonly 10 segments ≈ 14 KB) and doubles it every round trip until loss or a threshold.
  • Congestion avoidance — beyond the threshold, cwnd grows roughly one segment per RTT.
  • On loss (detected by duplicate ACKs) — cwnd is cut sharply (Reno halves it; CUBIC, Linux's default, reduces by about 30% and regrows along a cubic curve). On a timeout, cwnd collapses back to a minimum and slow start begins again.
  • BBR (Google) models bottleneck bandwidth and RTT instead of reacting to loss — much better on lossy, long-distance links. Enable with sysctl net.ipv4.tcp_congestion_control=bbr (with the fq qdisc).

Consequences you should be able to explain:

Enlarge diagramThroughput ≤ window ÷ round-trip time. To fill 1 Gbps over a 100 ms path you need about 12.5 MB in flight (thebandwidth-delay product).cwndtime (round trips) →slow start: ×2 per round tripcongestion avoidanceloss: cutloss: cut
Figure 2. The congestion window over time: it doubles each round trip in slow start, then grows slowly. Each loss cuts it, which is why a little packet loss hurts throughput so much.
Swipe sideways to see all of it, or tap to open it full screen.
  • Even 1% packet loss can cut TCP throughput dramatically on high-latency paths, because each loss shrinks cwnd and recovery takes many RTTs.
  • New connections are slow at first (slow start) — one reason connection reuse, keep-alive and HTTP/2 multiplexing matter.
  • Throughput ≤ window / RTT. This is the bandwidth-delay product (BDP): to fill a 1 Gbps link with 100 ms RTT you need ~12.5 MB in flight. Small buffers or windows cap throughput no matter how big the pipe — a classic cause of slow cross-region transfers (Mumbai ↔ US). Fix: larger socket buffers (net.core.rmem_max, net.ipv4.tcp_rmem), parallel streams, or tools designed for it (S3 multipart, aws s3 cp concurrency).

Nagle, delayed ACKs and latency

  • Nagle's algorithm batches small writes into fewer segments while data is unacknowledged — efficient, but adds latency for chatty request/response protocols.
  • Delayed ACK — receivers wait briefly (up to ~40 ms on Linux) to piggyback ACKs.
  • Together they can cause mysterious ~40 ms stalls. Latency-sensitive apps set TCP_NODELAY (most HTTP clients, databases and RPC frameworks already do).

Keepalive and idle timeouts

TCP keepalive sends probes on idle connections to detect dead peers and keep NAT/firewall/load-balancer state alive. Linux defaults are very long (first probe after 2 hours), so apps usually configure their own or rely on application-level pings.

Middleboxes drop idle connections:

Device Idle timeout
AWS ALB 60 s default (configurable)
AWS NLB 350 s for TCP
AWS NAT gateway 350 s
Many corporate firewalls Minutes

A long-lived connection (DB pool, message consumer) that sits idle longer than that is silently dropped; the next use fails with a reset or timeout. Fix: keepalives below the shortest timeout, pool max-idle settings, and retries.

Enlarge diagramApp poolNAT gatewayidle timeout 350 sDatabaseconnection idle for 10 minutes: NAT silently forgets itFix: TCP keepalive or pool max-idle below 350 squery, worksnext query on the old connectionreset or timeout
Figure 3. Why idle pooled connections fail: a NAT gateway or load balancer forgets connections that stay quiet too long. Keepalives shorter than the timeout keep them alive.
Swipe sideways to see all of it, or tap to open it full screen.

Head-of-line blocking

Because TCP delivers bytes strictly in order, one lost segment holds up all later data until it's retransmitted, even if later segments already arrived. HTTP/2 multiplexes many streams over one TCP connection, so a single loss stalls every stream — the motivation for QUIC.

Enlarge diagramHTTP/2 OVER TCP · ONE ORDERED BYTE STREAMHTTP/3 OVER QUIC · INDEPENDENT STREAMSsegment 1deliveredsegment 2deliveredsegment 3lostsegment 4waitingsegment 5waitingsegment 6waitingdelivereddelivereddelivereddelivereddeliveredlostdelivereddeliveredstream Astream BOnly stream B waits for its retransmission; stream A carries on.
Figure 4. Head-of-line blocking: TCP hands data over strictly in order, so one lost segment stalls everything behind it. QUIC keeps streams independent.
Swipe sideways to see all of it, or tap to open it full screen.

TCP Fast Open

Lets a client send data inside the SYN on repeat connections (using a cookie from a previous connection), saving one round trip. Limited deployment because middleboxes interfere.