Opscraft
DNS TTL & caching · Lesson 6 of 8 · 3 min read

TTL, failover and load balancing on AWS

DNS failover is bounded by TTL

Route 53 failover routing flips the answer when a health check fails (after the check's failure threshold — e.g. 3 × 30 s). Then clients switch only when their cached answer expires:

  • With TTL 300, clients may keep using the failed endpoint for up to 5 minutes after Route 53 switches — plus detection time.
  • With TTL 60, about a minute.

So failover records should use ~60 s TTLs, health checks with fast intervals (10 s) and sensible thresholds, and clients that retry and re-resolve on connection failure.

Enlarge diagramTTL 300detect ~90 sclients wait up to 300 s≈ 6 min 30 sTTL 60detect ~90 sclients wait up to 60 s≈ 2 min 30 sFailover records: TTL around 60 s, fast health checks, and clients that retry and re-resolve.
Figure 1. How long DNS failover really takes: detection by health checks, then the clients' cached TTL on top.
Swipe sideways to see all of it, or tap to open it full screen.

Why ELB IPs change and why aliases matter

Load balancers scale out and replace their nodes, so the IP addresses behind an ALB's DNS name change over time. Two consequences:

  1. Never hardcode an ALB's IPs in an A record — use an alias (Route 53 tracks the changes) or a CNAME to the ELB name.
  2. Clients that cache resolved IPs too long (JVM -1, long-lived connections that never reconnect) eventually talk to IPs that no longer belong to your load balancer → failures after the LB scales.

DNS switch vs load balancer switch

For blue-green cutovers and rollbacks, switching at the load balancer (listener rules, weighted target groups) takes effect in seconds for all clients. A DNS switch waits on caches and TTL-ignoring clients, and rollback waits again. DNS-based weighted routing also shifts traffic slowly and unevenly because a few large resolvers serve many users and all their users move together.

When you need fast global failover without DNS caching delays, AWS Global Accelerator gives static anycast IPs and shifts traffic between regional endpoints in seconds.

Enlarge diagramSwitch in DNSSwitch at the load balancerClients move as their caches expireStragglers that ignore TTL keep the old targetRight for region or provider movesListener rule or target-group weights changeTakes effect in seconds for every clientRight for blue/green and fast rollback
Figure 2. For cutovers and rollbacks, switching at the load balancer is instant for everyone; switching DNS waits for caches.
Swipe sideways to see all of it, or tap to open it full screen.

Lessons from incidents

"We changed DNS during the incident but users kept failing for 30 minutes" is a common postmortem finding. Action items: inventory critical records and pre-set appropriate low TTLs, document cutover steps, and prefer LB-level or Global Accelerator failover for critical paths.