Opscraft
Networking fundamentals · Lesson 2 of 9 · 3 min read

Layer 2: Ethernet, MAC addresses and switching

MAC addresses and frames

A MAC address (48 bits, e.g. 0a:1b:2c:3d:4e:5f) identifies a network interface on the local link. Ethernet frames carry source and destination MACs. MACs never cross a router — at every hop, the router builds a new frame.

Switches

A switch learns which MAC lives on which port (its MAC table) and forwards frames only to the right port; unknown destinations and broadcasts (ff:ff:ff:ff:ff:ff) are flooded to all ports in the broadcast domain (the VLAN).

VLANs split one physical switch into separate broadcast domains — the on-prem equivalent of separating subnets.

ARP

To send an IP packet to a host on the same subnet (or to the gateway), a machine must know its MAC. ARP broadcasts "who has 10.0.1.1?" and the owner replies with its MAC; results are cached (ip neigh). Duplicate IPs cause ARP conflicts — intermittent connectivity as two machines answer.

Enlarge diagramYour server10.0.1.15SwitchGateway10.0.1.1Other host10.0.1.20"Who has 10.0.1.1?"(broadcast)flooded to all…and herenot me: ignore"10.0.1.1 is at 0a:1b:2c:…"caches it (ip neigh)
Figure 1. ARP: to send a packet on the local network, a host must learn the MAC address that owns the IP. It shouts, and only the owner answers.
Swipe sideways to see all of it, or tap to open it full screen.

In an AWS VPC there's no real broadcast; the VPC fabric answers ARP itself, and gratuitous ARP/IP takeover tricks used on-prem for failover don't work — use Elastic IPs, ENI moves or load balancers instead.

MTU

The Maximum Transmission Unit is the largest frame payload on a link — 1500 bytes on standard Ethernet. AWS supports jumbo frames (9001 MTU) inside a VPC, but traffic leaving the VPC (internet, VPN, peering in some cases, Transit Gateway at 8500) uses smaller MTUs. Tunnels and overlays (VXLAN, IPsec, GRE) add headers that reduce effective MTU.

MTU problems have a signature: small requests work, large transfers hang (SSH connects but ls of a big directory stalls; HTTPS handshake succeeds but downloads freeze). Cause: large packets dropped and the ICMP "fragmentation needed" message blocked, breaking Path MTU Discovery. Fixes: allow ICMP type 3 code 4, lower the MTU, or MSS clamping on tunnel endpoints.

Enlarge diagramServer sends bigpackets9001-byte jumboframes inside the VPCSmaller link on thepathinternet or VPN: MTU1500Packet droppedrouter replies"fragmentationneeded" (ICMP)ICMP blockedthe sender nevershrinks its packetsFixAllow ICMP "fragmentation needed" through firewalls, or clamp the TCP MSS on the VPN or tunnel. Test with ping-M do -s 1472 host.
Figure 2. The MTU black hole: big packets are dropped on a smaller link, and the warning message that should come back is blocked. Small requests work; big transfers hang.
Swipe sideways to see all of it, or tap to open it full screen.