Opscraft
How DNS works behind the scenes · Lesson 7 of 9 · 5 min read

DNS in AWS VPCs and Kubernetes

The VPC resolver

Every VPC has the Route 53 Resolver at the VPC CIDR base +2 (e.g. 10.0.0.2) and at 169.254.169.253. It resolves private hosted zones, EC2 internal names, endpoint private DNS, forwarding rules, and public DNS. Limits matter at scale: each ENI can send up to 1,024 packets per second to it; beyond that, queries are dropped — visible as intermittent DNS timeouts on busy instances and Kubernetes nodes (where all pods on the node share that ENI's budget through CoreDNS).

Enlarge diagramVPCEC2 / podsRoute 53 Resolverbase + 2 or169.254.169.253outbound endpointforward rulesinbound endpointon-prem asks inon-premises DNScorp.example.comanswers: private hostedzones, EC2 names, public
Figure 1. The Route 53 Resolver inside every VPC, and the endpoints that connect it to on-premises DNS in both directions.
Swipe sideways to see all of it, or tap to open it full screen.

CoreDNS in Kubernetes

Pods' /etc/resolv.conf (set by the kubelet for dnsPolicy: ClusterFirst):

nameserver 172.20.0.10                      ← kube-dns Service (CoreDNS)
search shop.svc.cluster.local svc.cluster.local cluster.local ap-south-1.compute.internal
options ndots:5

The CoreDNS configuration (coredns ConfigMap):

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {   # answer Service/pod names from the API
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153                                    # metrics
    forward . /etc/resolv.conf                          # everything else → node's resolver (VPC resolver)
    cache 30                                            # cache up to 30 s
    loop
    reload
    loadbalance
}
corp.example.com:53 {                                   # added: forward an on-prem domain
    forward . 10.200.0.10 10.200.0.11
}
  • Names under cluster.local come from the Kubernetes API (Services, headless Service pods).
  • A short name like orders resolves via the search list to orders.shop.svc.cluster.local — only in the pod's own namespace; other namespaces need orders.payments or the FQDN.
Enlarge diagrampodresolv.conf →172.20.0.10CoreDNScache 30 sKubernetes API*.cluster.local: Services and podson-prem DNScorp.example.com (forward rule)VPC resolver (.2)private zones, EC2 names, internet
Figure 2. How a pod resolves names in EKS. Cluster names are answered by CoreDNS from the Kubernetes API; everything else is forwarded to the VPC resolver.
Swipe sideways to see all of it, or tap to open it full screen.

The ndots:5 multiplier

With ndots:5, a lookup of api.github.com (2 dots < 5) tries the search domains first:

Enlarge diagramapi.github.com.shop.svc.cluster.localapi.github.com.svc.cluster.localapi.github.com.cluster.localapi.github.com.ap-south-1.compute.internalapi.github.comFix: end external names with a dot (api.github.com.), lower ndots for the pod, or run NodeLocal DNSCache onevery node.NXDOMAINNXDOMAINNXDOMAINNXDOMAINanswer
Figure 3. With ndots:5, a name with fewer than five dots is first tried with every search suffix. One external lookup becomes five attempts, each for A and AAAA.
Swipe sideways to see all of it, or tap to open it full screen.

Each of those is done for both A and AAAA — up to 10 queries for one external name. Multiply by every outbound call in a busy cluster and CoreDNS and the VPC resolver's per-ENI limits get hammered.

Mitigations:

  • Use FQDNs with a trailing dot for external names in config (api.github.com.).
  • Lower ndots per pod:
spec:
  dnsConfig:
    options:
    - { name: ndots, value: "2" }
  • Deploy NodeLocal DNSCache — a caching DNS agent on every node (DaemonSet on a link-local IP) that absorbs repeated lookups, reduces conntrack pressure, and avoids the UDP conntrack race behind random 5-second delays.
  • Scale CoreDNS (cluster-proportional autoscaler or HPA), give it adequate CPU, spread replicas across nodes/AZs.

Debugging DNS in EKS

  1. From a debug pod: kubectl run -it --rm dns --image=nicolaka/netshoot -- bash, then dig orders.shop.svc.cluster.local, dig api.github.com, cat /etc/resolv.conf.
  2. CoreDNS health: kubectl -n kube-system get pods -l k8s-app=kube-dns, logs (enable the log plugin briefly), Prometheus metrics (coredns_dns_request_duration_seconds, coredns_forward_*, SERVFAIL counts), CPU throttling.
  3. Upstream: does the VPC resolver answer from the node (dig @169.254.169.253)? Any per-ENI throttling (linklocal_allowance_exceeded in ethtool -S)?
  4. Conntrack: conntrack -S insert failures on nodes → 5-second race.
  5. NetworkPolicies blocking egress to CoreDNS on UDP/TCP 53.
Enlarge diagramFrom a debugpoddig a Service andan externalname; checkresolv.confCoreDNSpods healthy?errors, latencymetricsUpstreamdoes the VPCresolver answerfrom the node?throttling?Conntrackinsert failures →the 5 s raceNetworkPolicyegress toCoreDNS on UDPand TCP 53allowed?12345
Figure 4. Debugging DNS in a cluster, from the pod outwards.
Swipe sideways to see all of it, or tap to open it full screen.