Deployment and Migrations
Shipping changes without downtime — rolling, blue-green and canary releases, feature flags, and the expand-contract pattern for schema changes
A design is not finished until you can change it. How you deploy decides how quickly you can fix a mistake, which matters more than how rarely you make one.
Release Strategies
| Strategy | How | Rollback speed | Cost |
|---|---|---|---|
| Recreate | Stop old, start new | Slow — redeploy | Downtime |
| Rolling | Replace instances a few at a time | Medium | None |
| Blue-green | Two full environments, switch traffic | Instant | Double infrastructure |
| Canary | Send 1% of traffic to the new version, then grow | Fast | Small |
Canary is usually the best default
It limits damage — a broken release affects 1% of users, not everyone. It gives real signal, because production traffic finds problems staging never will. And rollback is a routing change rather than a rebuild.
Blue-green also rolls back instantly, but needs a full duplicate environment and switches everyone at once, so a subtle bug hits 100% of users the moment you flip.
Feature Flags
Deploying code and releasing a feature are separate acts. A flag lets you ship dormant code and turn it on later — and off again without a deploy.
| Use | Why |
|---|---|
| Gradual rollout | Enable for 1%, then 10%, then everyone |
| Kill switch | Disable a misbehaving feature in seconds |
| Trunk-based development | Merge unfinished work safely behind a flag |
| Per-customer enablement | Enterprise features, betas |
Flags are debt with an expiry date
Every flag doubles the paths through the code. Ten flags mean up to 1,024 combinations, of which you test perhaps three.
Old flags are worse than useless: nobody remembers what they do, and nobody dares remove them.
Give each flag an owner and a removal date when it is created. Delete it once the feature is fully rolled out. A flag that has been at 100% for six months is dead code holding a live branch.
Database Migrations
This is where deployments actually go wrong, because a schema change and a code change cannot happen at the same instant.
During any rolling deploy, both versions run at once
For a period, old code and new code both talk to the same database.
That means the schema must work with both. A migration that drops a column the old code still reads takes down every instance that has not been replaced yet.
Expand and contract
The only safe way to change a schema without downtime. Three deploys instead of one.
Goal: rename `username` to `handle`
1. EXPAND add `handle`; code writes BOTH, reads `username`
2. BACKFILL copy existing values into `handle`
3. MIGRATE code reads `handle`, still writes both
4. CONTRACT stop writing `username`, then drop the columnAt every step, both the currently deployed version and the one being rolled out work correctly.
| Change | Safe in one step? |
|---|---|
| Add a nullable column | Yes |
| Add an index | Usually — create it concurrently |
| Add a table | Yes |
| Rename a column | No — expand/contract |
| Drop a column | No — stop reading it first, then drop |
Add NOT NULL | No — backfill first, then constrain |
| Narrow a column type | No — risks data loss |
Locking migrations on a large table cause outages
ALTER TABLE on a table with hundreds of millions of rows can hold a lock for minutes, and every query queues behind it. The application appears completely down.
Use the concurrent or online variants your database offers, do backfills in batches with pauses, and run them outside peak hours. A backfill that updates ten million rows in one statement is an outage waiting for a bad moment.
Rollback
Every deploy needs an answer to "how do we undo this?" before it starts.
| Component | Rollback |
|---|---|
| Stateless service | Easy — deploy the previous version |
| Feature behind a flag | Easiest — flip the flag |
| Additive schema change | Easy — the old code ignores the new column |
| Destructive schema change | Very hard — the data may be gone |
| A published message format | Hard — consumers already read it |
This is the real reason to prefer additive changes
Adding a column, a field or an endpoint is reversible: the previous version simply ignores it.
Dropping or renaming is not. Once a column is gone, rolling back the code does not bring the data back.
So: add now, remove much later, after the new version has been stable long enough that you will never roll back past it.
A Deployment Checklist
□ Can this be rolled back? How fast?
□ Do old and new code both work against this schema?
□ Is the risky part behind a feature flag?
□ Does the migration lock a large table?
□ Are backfills batched?
□ Which metrics indicate this release is bad?
□ Who is watching, and for how long?
□ Are health checks accurate enough for the balancer to spot failure?Related Reading
- Observability — the metrics that tell you a canary is failing
- Reliability — health checks and graceful degradation
- Services and Boundaries — expand-contract across service contracts
- Databases — indexes and the cost of adding them