Opscraft
Kubernetes troubleshooting · Lesson 1 of 11 · 4 min read

A troubleshooting method

This topic is Kubernetes on the job: a pod won't start, a Service returns 503, a node goes NotReady, a deploy is stuck, a namespace won't delete. Knowing the commands isn't enough; senior interviews want a systematic method, the ability to decode every common status and error message quickly, and incident stories told well. Everything here is organised by symptom, with the exact commands and the usual causes in order of likelihood, plus debugging for Flux-based GitOps setups.

Work from symptom to layer

Most Kubernetes problems live in one of these layers:

Enlarge diagramGitOps / CIdid the change evenarrive?API + admissionwas it accepted?Schedulingdid it get a node?Node + runtimedid the container start?Appdoes it keep running?Probes + endpointsis it receiving traffic?Service, DNS,NetworkPolicycan others reach it?Ingress / loadbalancercan users reach it?12345678
Figure 1. Where Kubernetes problems live. Work down the layers until you find where the request or the rollout stops.
Swipe sideways to see all of it, or tap to open it full screen.

Ask in order: What changed? (deploys, config, node/cluster upgrades, traffic) What's the scope? (one pod, one node, one namespace, everything) Where does the request stop?

The core loop

kubectl get pods -n shop -o wide                     # status, restarts, node, IP
kubectl describe pod orders-7d9f-abc -n shop         # spec, state, last state, exit code, EVENTS
kubectl logs orders-7d9f-abc -n shop -c orders       # current container logs
kubectl logs orders-7d9f-abc -n shop --previous      # the crashed instance's logs
kubectl get events -n shop --sort-by=.lastTimestamp  # recent namespace events
kubectl get events -A --field-selector type=Warning  # warnings cluster-wide

describe + --previous logs solve the majority of pod problems. Events are kept only about one hour by default, so look early (or ship events to your logging system with an event exporter).

Enlarge diagramget pods -o widestatus, restarts, nodedescribe podlast state, exit code,EVENTSlogs --previouswhy the last run diedget eventssorted by time; kept~1 hour1234
Figure 2. The core loop that solves most pod problems.
Swipe sideways to see all of it, or tap to open it full screen.

More tools

kubectl get pod orders-7d9f-abc -o yaml              # full spec and status (conditions, containerStatuses)
kubectl get deploy,rs,pods -l app=orders -n shop     # whole ownership chain
kubectl rollout status deploy/orders -n shop
kubectl top pods -n shop --containers                # usage (needs metrics-server)
kubectl top nodes
kubectl exec -it orders-7d9f-abc -n shop -- sh       # shell (if the image has one)
kubectl port-forward svc/orders 8080:80 -n shop      # test a Service from your laptop
kubectl auth can-i get secrets --as=system:serviceaccount:shop:orders -n shop
kubectl get --raw /readyz?verbose                    # API server health

Debugging minimal images and nodes

Distroless images have no shell. Use ephemeral debug containers:

kubectl debug -it orders-7d9f-abc -n shop --image=nicolaka/netshoot --target=orders
# shares the target container's process namespace: ps, curl, dig, tcpdump, ss
kubectl debug orders-7d9f-abc -n shop -it --copy-to=orders-debug --container=orders -- sh   # copy of the pod with changes
kubectl debug node/ip-10-20-1-15.ap-south-1.compute.internal -it --image=ubuntu
# host filesystem at /host: chroot /host; journalctl -u kubelet; crictl ps

A throwaway network test pod is invaluable:

kubectl run -it --rm netshoot -n shop --image=nicolaka/netshoot -- bash
# dig, curl, nc, mtr, tcpdump from inside the cluster network
Enlarge diagramPOD ORDERS-7D9Forders (distroless)no shell, no curldebug: netshootps, curl, dig, tcpdump, ssshares
Figure 3. Debugging a distroless container: kubectl debug adds a temporary container with tools that shares the target's processes and network.
Swipe sideways to see all of it, or tap to open it full screen.