Opscraft
Deployment strategies · Lesson 7 of 10 · 3 min read

Database and stateful changes

The fundamental problem

With rolling, canary and blue-green, old and new code run against the same database at the same time — and during a rollback, old code runs against a schema the new version may have changed. A destructive migration (dropping or renaming a column) breaks whichever version doesn't expect it.

Expand/contract (parallel change)

Example: rename customers.fullname to customers.display_name.

Enlarge diagramExpandadd display_name(nullable)Dual writewrite both, read newwith fallback; backfillSwitch readsread onlydisplay_name, stillwrite bothContractstop writing fullname,then drop it1234
Figure 1. Expand and contract: rename a column in four safe steps, so any single deploy can be rolled back.
Swipe sideways to see all of it, or tap to open it full screen.
  1. Expand — add display_name (nullable). Old code ignores it. Deploy freely.
  2. Migrate code (dual write) — new version writes both columns and reads display_name with fallback to fullname. Backfill existing rows in batches.
  3. Switch reads — once backfilled, read only display_name; still write both (keeps rollback possible).
  4. Contract — after a release or two, when no running or rollback-able version uses fullname, stop writing it and finally drop it.

Each step is backward compatible with the previous version, so any single deploy can be rolled back safely.

Rules of thumb:

  • Additive changes are safe; destructive ones come last, separately.
  • Make new columns nullable or with defaults; avoid long table locks (online schema change tools for MySQL like gh-ost/pt-osc; CREATE INDEX CONCURRENTLY in Postgres).
  • Run migrations as a separate, idempotent pipeline step or Kubernetes Job before the app rollout — not from every pod at startup (races, many pods migrating simultaneously).
  • Version migrations with Flyway/Liquibase/Alembic/golang-migrate.
  • Take a snapshot before risky migrations; know whether "rollback" means a down-migration or a restore.

Other compatibility surfaces

  • APIs — additive changes only; breaking changes get a new version (/v2) served alongside the old until clients migrate.
  • Messages/events — consumers must tolerate unknown fields; use schema registries with compatibility rules (Avro/Protobuf).
  • Caches — new code reading old cache entries (and vice versa); version cache keys.
  • Config/feature flags — defaults safe for both versions.