Opscraft
Kubernetes troubleshooting · Lesson 9 of 11 · 3 min read

Cluster-level problems

Things broke after a cluster upgrade

Most often: manifests or Helm charts using removed API versions (e.g. policy/v1beta1 PodDisruptionBudget, autoscaling/v2beta2 HPA, old flowcontrol APIs). Apply fails with no matches for kind ... in version .... Check before upgrading with kubent/pluto, EKS upgrade insights, and API server deprecated-API metrics; update manifests and chart versions. Also add-on version compatibility (CNI, CoreDNS, kube-proxy, controllers).

Namespace stuck in Terminating

Resources inside still have finalizers whose controller is gone (an uninstalled operator/CRD, a load balancer controller removed before its Ingresses):

kubectl get namespace shop -o json | jq '.status.conditions'
kubectl api-resources --verbs=list --namespaced -o name | xargs -n1 kubectl get -n shop --ignore-not-found --show-kind

Fix the controller (reinstall it so it can clean up) or, as a last resort and knowing the external resources may be orphaned, remove the finalizers from the stuck objects.

Enlarge diagramNamespaceTerminatingIngressfinalizer: LB controllerControlleruninstalled: nobodyremoves itwaits forwaits forFix the controller first. Removing finalizers by hand can orphan cloud resources.
Figure 1. A namespace stuck in Terminating: something inside has a finalizer whose controller is gone.
Swipe sideways to see all of it, or tap to open it full screen.

Admission webhooks blocking everything

Errors like failed calling webhook "validation.gatekeeper.sh": ... context deadline exceeded — a webhook with failurePolicy: Fail is down or slow. Deploys fail cluster-wide. Fix the webhook's pods; if they can't start because the webhook blocks them, temporarily exclude system namespaces or edit the webhook configuration. Prevent with HA webhooks, short timeouts and namespace exemptions.

Enlarge diagramkubectl applyany deployAPI servercalls the webhookwebhook podsdown: timeouttimeoutFix the webhook pods; exclude kube-system and the webhook's own namespace from it.
Figure 2. A down admission webhook with failurePolicy: Fail blocks every matching create, even the webhook's own pods.
Swipe sideways to see all of it, or tap to open it full screen.

RBAC "forbidden"

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:shop:orders" cannot list resource "pods" ... — grant the needed verbs via Role/RoleBinding (least privilege). Test with kubectl auth can-i. On EKS, human access problems are usually access entries (or aws-auth on older clusters).

Certificates and control plane

Self-managed clusters: expired component certificates (kubeadm defaults to 1 year) → API server refusing connections; etcd disk latency → slow API; check kubectl get --raw /readyz?verbose. On EKS the control plane is managed — look at control plane logs in CloudWatch and AWS Health events.