Opscraft
Amazon EKS · Lesson 8 of 11 · 3 min read

Upgrades and day-2 operations

In-place upgrade procedure

Kubernetes upgrades go one minor version at a time (1.29 → 1.30 → 1.31 — two separate upgrades).

  1. Prepare: read the Kubernetes and EKS release notes; check EKS upgrade insights and tools like kubent/pluto for deprecated/removed APIs in manifests and Helm charts; check add-on and controller compatibility (LB controller, Karpenter, Flux, Istio...); verify PDBs won't block drains; test in a non-prod cluster first.
  2. Upgrade the control plane (aws eks update-cluster-version / the EKS Terraform module's kubernetes_version, called cluster_version before v21). Takes ~10–20 minutes, API stays available.
  3. Upgrade managed add-ons: VPC CNI, CoreDNS, kube-proxy, EBS CSI, Pod Identity Agent — to versions matching the new Kubernetes version.
  4. Upgrade nodes: managed node groups (rolling replacement respecting PDBs), Karpenter (update the AMI alias/selector → drift replaces nodes within disruption budgets), Fargate (restart pods).
  5. Upgrade other controllers if needed, then verify workloads and monitoring.

Version skew: nodes can be up to three minor versions behind the control plane, but don't let them drift — they also need upgrading before the next control plane jump.

Enlarge diagramPreparerelease notes,upgrade insights,removed APIsControl planeabout 10–20min, API stays upAdd-onsCNI, CoreDNS,kube-proxy, CSINodesrolling replace,PDBs respectedVerifyworkloads andmonitoring12345
Figure 1. An in-place EKS upgrade, one minor version at a time.
Swipe sideways to see all of it, or tap to open it full screen.

Blue/green cluster upgrades

For major changes (several versions behind, CNI changes, risky platform changes): build a new cluster at the target version with Terraform, bootstrap Flux pointing at the same Git repo so all apps deploy identically, validate, then shift traffic gradually (Route 53 weighted records or ALB/Global Accelerator weights), and retire the old cluster. Pod Identity makes this easier because IAM roles don't need per-cluster OIDC trust changes. Stateful workloads need data migration planning.

Enlarge diagramGit reposame manifestsblue cluster 1.30Fluxgreen cluster 1.33FluxRoute 53weights 90 / 10
Figure 2. Blue/green cluster upgrade: build a new cluster, let Flux deploy the same Git state, then shift traffic gradually.
Swipe sideways to see all of it, or tap to open it full screen.

Other routine operations

  • Node AMI patching via node group updates or Karpenter drift/expiry.
  • Certificate rotation for webhooks/controllers (cert-manager).
  • Capacity reviews: requests vs usage, node utilisation, IP availability per subnet.
  • Backup of cluster resources with Velero (objects + EBS snapshots to S3) — plus GitOps as the primary rebuild mechanism.