Designing for reliability
Resilience patterns
| Pattern | Purpose | Notes |
|---|---|---|
| Timeouts | Don't wait forever on dependencies | Every network call; shorter than the caller's timeout |
| Retries with exponential backoff and jitter | Survive transient failures | Only for idempotent operations; limit attempts; retry budgets to avoid retry storms |
| Circuit breakers | Stop calling a failing dependency, fail fast, recover gradually | Resilience4j, Envoy/Istio outlier detection |
| Bulkheads | Isolate resources per dependency/tenant so one can't exhaust all | Separate thread/connection pools |
| Rate limiting / load shedding | Protect the system under overload; reject low-priority work first | Return 429/503 fast rather than collapse |
| Graceful degradation | Serve partial functionality (cached recommendations, read-only mode) | Feature flags |
| Idempotency keys | Safe retries for payments/orders | Store request IDs |
| Queues | Absorb bursts, decouple producers and consumers | Watch age of oldest message |
In Kubernetes: readiness/liveness/startup probes (liveness must not depend on downstream services), PodDisruptionBudgets, topology spread across AZs, resource requests and limits, HPA/KEDA, graceful shutdown (preStop, terminationGracePeriodSeconds), multiple replicas.
Safe change management
Most incidents are caused by changes. Reduce risk with: small, frequent deployments; canary or blue/green with automated analysis; feature flags to decouple deploy from release; automated rollback on SLO regression; change freezes around peak events (sale days) based on error budget; config changes going through the same pipeline and validation as code.
Capacity planning and load testing
- Forecast from organic growth + launches/sales events (e.g. festival sales in India drive huge spikes).
- Know each service's saturation point via load tests (k6, Gatling, Locust, JMeter) in a production-like environment; keep headroom (e.g. N+1 AZ capacity — survive losing one AZ).
- Pre-scale before known events; check quotas (EC2 vCPU limits, IP addresses, NAT throughput, database connections, API rate limits).
Chaos engineering
Run controlled experiments to verify resilience assumptions: define steady state (SLIs), hypothesise ("losing one AZ won't breach the SLO"), inject failure (kill pods/nodes, add latency, fail an AZ) with AWS Fault Injection Service, Chaos Mesh or LitmusChaos, limit blast radius, observe, fix weaknesses. Start in staging, graduate to production game days with stop conditions.