Opscraft
Kubernetes troubleshooting · Lesson 3 of 11 · 5 min read

Pods that crash or restart

CrashLoopBackOff

The container starts and exits repeatedly; Kubernetes backs off restarts (10 s, 20 s, 40 s... up to 5 min). It's a symptom, not a cause:

kubectl logs orders-7d9f-abc -n shop --previous           # why did the LAST run die?
kubectl describe pod orders-7d9f-abc -n shop              # Last State: Terminated, Reason, Exit Code
kubectl get pod orders-7d9f-abc -n shop -o jsonpath='{.status.containerStatuses[0].lastState}'
Enlarge diagram✕✕✕✕10 s20 s40 s… up to 5 minkubectl logs <pod> --previous · describe: Last State, Reason, Exit Code
Figure 1. CrashLoopBackOff is Kubernetes waiting longer between restarts. The cause is in the previous container's logs and exit code.
Swipe sideways to see all of it, or tap to open it full screen.

Exit codes

Exit code Meaning Common causes
0 Exited successfully The main process finished — wrong command, or a one-off task run as a Deployment (use a Job)
1 Application error Exception at startup: bad config, missing env var, can't reach a dependency, migration failure
2 Misuse of shell builtins / bad arguments Wrong command-line flags
126 Command not executable Missing execute permission
127 Command not found Wrong command/entrypoint path, binary missing in slim image
128+n Killed by signal n —
137 128+9 SIGKILL OOMKilled (check Reason: OOMKilled), or killed after the termination grace period, or by liveness probe failure → kubelet restart
139 128+11 SIGSEGV Segfault — native code bug, arch/libc mismatch
143 128+15 SIGTERM Graceful termination (normal during rollouts/scale-down)
Enlarge diagram01127137139143finished: should it be a Job?app error at startupcommand not foundSIGKILL: OOMKilled or forcedsegfaultSIGTERM: normal shutdown
Figure 2. Exit codes decoded. Above 128 means "killed by signal (code − 128)".
Swipe sideways to see all of it, or tap to open it full screen.

OOMKilled

Reason: OOMKilled, exit 137 — the container exceeded its memory limit (cgroup OOM), even if the node has free memory.

Enlarge diagramLEAK: STEADY CLIMBSPIKE: SUDDEN JUMPlimitlimit
Figure 3. Two memory shapes behind OOMKilled. A steady climb is a leak; a sudden jump is one big piece of work.
Swipe sideways to see all of it, or tap to open it full screen.
  1. Look at memory over time (kubectl top, Prometheus container_memory_working_set_bytes): a steady climb → leak; a sudden spike → a large request, batch, or cache load.
  2. Compare to the limit. For JVMs: heap + metaspace + threads + direct buffers must fit — use -XX:MaxRAMPercentage=75 rather than fixed -Xmx larger than the container.
  3. Fix: right-size limits from real usage (VPA recommendations help), fix leaks (heap dumps, profiling), stream instead of loading whole files, and alert at ~90% of limit.

Node-level OOM (the kernel killing processes because the node ran out of memory, often due to pods without limits or overcommitted limits) shows up in node dmesg and as kubelet evictions.

Liveness probes killing healthy apps

Events show Liveness probe failed: ... followed by Killing container. Causes: the probe hits a slow endpoint or one that checks dependencies (a database blip restarts every pod — a cascading failure), timeout too short (timeoutSeconds: 1 default) under load or GC pauses, or the app starts slowly and liveness begins before it's up.

Fixes: liveness should be a cheap "process is alive" check, not dependency checks; add a startup probe for slow starters (liveness waits until startup succeeds); raise timeoutSeconds/failureThreshold; use initialDelaySeconds only as a blunt fallback.

Other crash causes

  • Config/secrets: missing env vars, wrong DB host, secret value with a trailing newline.
  • Permissions: readOnlyRootFilesystem: true but the app writes to /tmp (mount an emptyDir); running as non-root but files owned by root in the image; volume ownership vs runAsUser → set fsGroup.
  • Dependencies at startup: the app exits if the DB isn't reachable — make it retry with backoff instead of crashing.
  • Wrong architecture: exec format error.