Networking: Services, DNS, policies, ingress
Service has no endpoints
kubectl get svc orders -n shop -o wide # selector
kubectl get pods -n shop --show-labels -l app=orders
kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders
Causes: selector doesn't match pod labels (typo, label changed in a refactor), pods not Ready, pods in another namespace. Also check the targetPort matches the container port (name or number) — a Service with endpoints but the wrong targetPort gives connection refused.
Testing connectivity step by step
From a netshoot pod in the same namespace:
dig orders.shop.svc.cluster.local # DNS → ClusterIP
curl -sv http://orders.shop.svc.cluster.local/health # via Service
curl -sv http://10.20.34.17:8080/health # directly to a pod IP (bypasses Service)
nc -zv orders-db.internal.example.com 5432 # external dependency
If pod-IP works but Service doesn't → Service/endpoints/kube-proxy. If neither works → app, NetworkPolicy, or security groups for pods.
DNS failures
- Short name fails across namespaces → use
svc.namespaceor FQDN. - Intermittent 5-second delays → UDP conntrack race; NodeLocal DNSCache.
- CoreDNS overloaded → ndots:5 amplification, scale CoreDNS, NodeLocal DNSCache.
- External names fail → CoreDNS forward to VPC resolver, VPC resolver per-ENI limits, egress NetworkPolicy blocking UDP/TCP 53.
NetworkPolicies
After introducing default-deny, things break in predictable ways: DNS (allow egress to kube-dns on UDP/TCP 53), traffic from the ingress controller namespace, Prometheus scraping, egress to AWS APIs/RDS (CIDR rules). Check which policies select a pod:
kubectl get networkpolicy -n shop
kubectl describe networkpolicy -n shop
And confirm the CNI enforces policies at all.
Ingress problems
| Symptom | Check |
|---|---|
| 404 from the controller | Host header matches host, path and pathType, ingressClassName, rewrite annotations, conflicting Ingresses |
| 503 from the controller | Backend Service has no ready endpoints, wrong service port |
| 502 | Pods resetting connections, wrong backend protocol (HTTP vs HTTPS), keep-alive mismatch |
| ALB not created (AWS LB Controller) | Controller logs; subnet tags (kubernetes.io/role/elb), IAM permissions, annotations, IngressClass |
| ALB targets unhealthy | Health check path/port, pod readiness, security groups, target-type |
Pods can't reach AWS services or the internet
Private subnets without NAT/endpoints, security groups (for pods), NACLs, IRSA/Pod Identity not configured (403 AccessDenied rather than network errors), SNAT settings in the VPC CNI.