Deployment strategies · Lesson 8 of 10 · 4 min read
Rollback, verification and multi-region rollouts
Verify every deployment
kubectl rollout status only proves pods are running. A deployment is successful when:
- Health and smoke tests pass (key endpoints, a synthetic transaction).
- SLO metrics — error rate, latency, saturation — stay within thresholds over a bake time.
- Business metrics don't drop.
- No new error signatures in logs/traces.
A pipeline that's green while production is broken is missing post-deploy verification.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Rollback design
- Keep previous artifacts and revisions — immutable image tags,
revisionHistoryLimit, Helm release history, Git history. - Automate it — Argo Rollouts/Flagger abort automatically; CodeDeploy rolls back on CloudWatch alarms; ECS circuit breaker; pipeline jobs that revert the GitOps commit.
- Make it boring — practise rollbacks; time them.
- Decide in advance — pre-agreed rollback triggers (error rate > X% for Y minutes). During an incident the on-call/incident lead rolls back first and investigates afterwards; you don't need root cause to roll back.
- Know when rollback isn't possible — irreversible data migrations, external side effects (emails sent, payments captured). Then roll forward with a fix, or use flags to disable the feature.
Rollback commands by tool:
kubectl rollout undo deploy/orders # plain Kubernetes (temporary under GitOps)
helm rollback orders 41 # Helm release revision
git revert <commit> && git push # GitOps — Flux reconciles the old state
flux suspend kustomization orders # emergency: stop reconciliation while you act
kubectl argo rollouts abort orders # Argo Rollouts
aws deploy stop-deployment --deployment-id d-XXXX --auto-rollback-enabled # CodeDeploy
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Multi-region and large fleets
- Deploy to one region (ideally the smallest or a canary region) first, bake, then the next — never all regions at once.
- Within a region, roll AZ by AZ or cell by cell for very large systems (cell-based architecture limits blast radius).
- Keep regions on compatible versions; a cross-region API or data replication format change must be backward compatible.
- Automate the wave plan in the pipeline with gates between waves.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.