Opscraft
SRE & incident management · Lesson 6 of 7 · 4 min read

Designing for reliability

Resilience patterns

Pattern Purpose Notes
Timeouts Don't wait forever on dependencies Every network call; shorter than the caller's timeout
Retries with exponential backoff and jitter Survive transient failures Only for idempotent operations; limit attempts; retry budgets to avoid retry storms
Circuit breakers Stop calling a failing dependency, fail fast, recover gradually Resilience4j, Envoy/Istio outlier detection
Bulkheads Isolate resources per dependency/tenant so one can't exhaust all Separate thread/connection pools
Rate limiting / load shedding Protect the system under overload; reject low-priority work first Return 429/503 fast rather than collapse
Graceful degradation Serve partial functionality (cached recommendations, read-only mode) Feature flags
Idempotency keys Safe retries for payments/orders Store request IDs
Queues Absorb bursts, decouple producers and consumers Watch age of oldest message
Enlarge diagramorderscallertimeoutnever wait foreverretry + jitteridempotent callsonlycircuit breakerfail fast when it isdowndepAlso: bulkheads (separate pools), load shedding (fast 429/503), graceful degradation, idempotency keys, queues.
Figure 1. Resilience patterns around a call to a dependency.
Swipe sideways to see all of it, or tap to open it full screen.

In Kubernetes: readiness/liveness/startup probes (liveness must not depend on downstream services), PodDisruptionBudgets, topology spread across AZs, resource requests and limits, HPA/KEDA, graceful shutdown (preStop, terminationGracePeriodSeconds), multiple replicas.

Enlarge diagram1 user requestgateway × 3 retries3 callsservice × 3 retries9 callsdatabase27 calls while alreadyslowRetry at one layer, with backoff, jitter and a retry budget.
Figure 2. Why naive retries are dangerous: every layer retrying multiplies load on the struggling service.
Swipe sideways to see all of it, or tap to open it full screen.

Safe change management

Most incidents are caused by changes. Reduce risk with: small, frequent deployments; canary or blue/green with automated analysis; feature flags to decouple deploy from release; automated rollback on SLO regression; change freezes around peak events (sale days) based on error budget; config changes going through the same pipeline and validation as code.

Capacity planning and load testing

  • Forecast from organic growth + launches/sales events (e.g. festival sales in India drive huge spikes).
  • Know each service's saturation point via load tests (k6, Gatling, Locust, JMeter) in a production-like environment; keep headroom (e.g. N+1 AZ capacity — survive losing one AZ).
  • Pre-scale before known events; check quotas (EC2 vCPU limits, IP addresses, NAT throughput, database connections, API rate limits).
Enlarge diagramAZ AAZ BAZ C50% of peakeach AZ sized for half50% of peakeach AZ sized for halfloststill OKAlso check quotas before big events: vCPUs, IPs, NAT throughput, DB connections.
Figure 3. Plan capacity so you can lose a whole AZ at peak and still meet the SLO.
Swipe sideways to see all of it, or tap to open it full screen.

Chaos engineering

Run controlled experiments to verify resilience assumptions: define steady state (SLIs), hypothesise ("losing one AZ won't breach the SLO"), inject failure (kill pods/nodes, add latency, fail an AZ) with AWS Fault Injection Service, Chaos Mesh or LitmusChaos, limit blast radius, observe, fix weaknesses. Start in staging, graduate to production game days with stop conditions.

Enlarge diagramSteady stateSLIs normalHypothesislosing one AZ willnot breach theSLOInjectFIS: kill nodes,add latencyObservedid SLIs hold?Fix + repeatsmall blast radiusfirst12345
Figure 4. A chaos experiment is a controlled test of a belief about your system.
Swipe sideways to see all of it, or tap to open it full screen.