Opscraft
Kubernetes troubleshooting · Lesson 6 of 11 · 5 min read

Networking: Services, DNS, policies, ingress

Service has no endpoints

kubectl get svc orders -n shop -o wide            # selector
kubectl get pods -n shop --show-labels -l app=orders
kubectl get endpointslices -n shop -l kubernetes.io/service-name=orders

Causes: selector doesn't match pod labels (typo, label changed in a refactor), pods not Ready, pods in another namespace. Also check the targetPort matches the container port (name or number) — a Service with endpoints but the wrong targetPort gives connection refused.

Enlarge diagramService ordersselector: app=orderspodslabels: app=order-apino matchAlso check: pods Ready? same namespace? targetPort = container port?
Figure 1. The classic "Service has no endpoints": the selector and the pod labels do not match, so the Service has nothing to send to.
Swipe sideways to see all of it, or tap to open it full screen.

Testing connectivity step by step

From a netshoot pod in the same namespace:

dig orders.shop.svc.cluster.local              # DNS → ClusterIP
curl -sv http://orders.shop.svc.cluster.local/health   # via Service
curl -sv http://10.20.34.17:8080/health        # directly to a pod IP (bypasses Service)
nc -zv orders-db.internal.example.com 5432     # external dependency

If pod-IP works but Service doesn't → Service/endpoints/kube-proxy. If neither works → app, NetworkPolicy, or security groups for pods.

Enlarge diagramdig the Servicenamefails → DNScurl the Servicefails but pod works →Service, endpointscurl the pod IPfails → app,NetworkPolicy,security groupnc the dependencyfails → egress, NAT,endpoints1234
Figure 2. Testing connectivity layer by layer from a debug pod. Where the chain breaks tells you which layer to fix.
Swipe sideways to see all of it, or tap to open it full screen.

DNS failures

  • Short name fails across namespaces → use svc.namespace or FQDN.
  • Intermittent 5-second delays → UDP conntrack race; NodeLocal DNSCache.
  • CoreDNS overloaded → ndots:5 amplification, scale CoreDNS, NodeLocal DNSCache.
  • External names fail → CoreDNS forward to VPC resolver, VPC resolver per-ENI limits, egress NetworkPolicy blocking UDP/TCP 53.

NetworkPolicies

After introducing default-deny, things break in predictable ways: DNS (allow egress to kube-dns on UDP/TCP 53), traffic from the ingress controller namespace, Prometheus scraping, egress to AWS APIs/RDS (CIDR rules). Check which policies select a pod:

kubectl get networkpolicy -n shop
kubectl describe networkpolicy -n shop

And confirm the CNI enforces policies at all.

Enlarge diagramDNSegress to kube-dns, UDP + TCP 53Ingress controllerfrom its namespace to the appPrometheusscraping the metrics portWebhooksAPI server to webhook pods
Figure 3. After adding default-deny, these are the flows teams forget to allow.
Swipe sideways to see all of it, or tap to open it full screen.

Ingress problems

Symptom Check
404 from the controller Host header matches host, path and pathType, ingressClassName, rewrite annotations, conflicting Ingresses
503 from the controller Backend Service has no ready endpoints, wrong service port
502 Pods resetting connections, wrong backend protocol (HTTP vs HTTPS), keep-alive mismatch
ALB not created (AWS LB Controller) Controller logs; subnet tags (kubernetes.io/role/elb), IAM permissions, annotations, IngressClass
ALB targets unhealthy Health check path/port, pod readiness, security groups, target-type
Enlarge diagramclientALB / controllerhost + path rulesServiceendpointspods404: no rule matched503: no ready endpoints502: bad reply
Figure 4. Who returned the error? 404 means no rule matched; 503 means no healthy backend; 502 means the backend misbehaved.
Swipe sideways to see all of it, or tap to open it full screen.

Pods can't reach AWS services or the internet

Private subnets without NAT/endpoints, security groups (for pods), NACLs, IRSA/Pod Identity not configured (403 AccessDenied rather than network errors), SNAT settings in the VPC CNI.