Consistency and CAP
What you must give up when data lives in more than one place, and how to choose which guarantee each feature needs
The moment your data exists on two machines, they can disagree. Everything on this page is about managing that fact.
The CAP Theorem
CAP says a distributed system can guarantee at most two of three properties:
| Letter | Property | Meaning |
|---|---|---|
| C | Consistency | Every read sees the most recent write |
| A | Availability | Every request gets an answer |
| P | Partition tolerance | The system keeps working when the network splits |
CAP is usually stated misleadingly
"Pick two" implies you have a free choice of three. You do not.
Network partitions happen. Cables are cut, switches fail, datacentres lose connectivity. You cannot opt out of P — you can only decide what to do when a partition occurs.
So the real choice is only ever between C and A, and only during a partition. When the network is healthy you can have both.
What the choice looks like
Two replicas, and the link between them breaks. A write arrives at one side.
| Choice | During a partition | Suits |
|---|---|---|
| CP — consistency | Refuse requests rather than risk stale data | Payments, inventory, bookings |
| AP — availability | Answer with possibly stale data | Feeds, likes, view counts, DNS |
The right answer differs per feature, not per company
The same product usually needs both.
A shop can serve a slightly stale product description — nobody is harmed by a 30-second-old description. The same shop must not serve a stale stock count at checkout, or it sells items it does not have.
Strong design says: "the catalogue is AP, the checkout path is CP." Choosing one guarantee for the entire system is almost always wrong.
Consistency Models
"Consistent" is not binary. There is a ladder, and each rung costs more than the one below.
| Model | Guarantee | Cost |
|---|---|---|
| Strong | Every read sees the latest write | Slowest — needs coordination |
| Read-your-writes | You see your own writes; others may lag | Cheap and usually enough |
| Monotonic reads | You never see time go backwards | Cheap |
| Eventual | Given no new writes, replicas converge | Fastest, weakest |
Eventual consistency has a sharp edge
"Eventually consistent" sounds harmless. The failure it permits is not:
Read 1 (replica A, caught up): balance = £100
Read 2 (replica B, lagging): balance = £150
Read 3 (replica A again): balance = £100The user watches their balance change and change back. Nothing is broken, no error is logged, and they will not trust the number again.
Monotonic reads fixes exactly this by pinning a user to one replica, and it is far cheaper than full strong consistency.
Read-your-writes is the one to reach for
Most "the data is wrong" complaints are really read-after-write problems: a user saves something, reads it back from a lagging replica, and their change appears to have vanished.
Full strong consistency would fix it, at a heavy cost. Read-your-writes fixes it cheaply:
- Route that user's reads to the primary for a few seconds after they write, or
- Keep the value in their session and serve it back to them, or
- Have the client hold the version it wrote and wait for a replica that has it.
ACID and BASE
Two philosophies, and the names are deliberately opposed.
| ACID (typically relational) | BASE (typically distributed) |
|---|---|
| Atomic — all or nothing | Basically Available |
| Consistent — constraints hold | Soft state — may be in flux |
| Isolated — transactions do not interfere | Eventually consistent |
| Durable — committed means saved |
Isolation levels are where real bugs live
Even a single relational database offers several isolation levels, and the default is often not the safest:
| Level | Prevents | Still allows |
|---|---|---|
| Read uncommitted | almost nothing | dirty reads |
| Read committed | dirty reads | non-repeatable reads |
| Repeatable read | non-repeatable reads | phantom rows |
| Serializable | everything | nothing — but it is slow |
PostgreSQL defaults to read committed. Two transactions reading the same row can therefore see different values, which breaks "check then act" patterns such as "is there stock left? then decrement it".
The fix is an explicit lock or an atomic conditional update — not a re-read.
Idempotency: The Practical Safeguard
Networks lose responses. The client retries. The action happens twice.
Client → POST /charge → server charges £50 → response LOST
Client retries → server charges £50 AGAINAn idempotent operation can be repeated safely. The client sends a unique key, and the server records which keys it has already handled.
POST /charge
Idempotency-Key: 7f3a-9b21
First time: perform the charge, store the result under that key
Retry: key already seen → return the stored result, charge nothingThis is worth mentioning in almost any design
Retries are unavoidable in a distributed system, so any operation that changes state needs an answer to "what happens if this arrives twice?"
Reads and deletes are naturally idempotent. Creates and increments are not, and those are exactly the ones that cost money when they double.
Distributed Transactions
Sometimes an action must span services — take payment and reserve stock. Two-phase commit exists but holds locks across the network and blocks if the coordinator dies.
The common alternative is a saga: a sequence of local transactions, each with a compensating undo.
1. reserve stock → ok
2. charge the card → ok
3. book the courier → FAILS
compensate backwards:
refund the card
release the stockSagas trade atomicity for availability
There is a window where stock is reserved and the card is charged but the courier is not booked. The system is temporarily inconsistent by design.
That is acceptable when the compensations are reliable and the window is short. It is not acceptable where an observer must never see the intermediate state.
The honest framing: "we use a saga, so there is a brief window where the order is partially complete; the compensating actions close it within seconds."
How to Decide
Ask two questions about each piece of data.
What happens if a reader sees data a few seconds old?
"A like count is briefly wrong" — eventual consistency is fine.
"We sell stock we do not have" — you need strong consistency on that path.
What happens if the operation runs twice?
"The same row is set to the same value" — naturally idempotent.
"The customer is charged twice" — you need an idempotency key.
Answer those two per feature, and the consistency design follows.
Related Reading
- Databases — replication lag, the source of most inconsistency
- Caching — a cache is deliberate, bounded staleness
- Queues and Async Work — delivery guarantees and why idempotency matters there too
- Reliability — retries, which are what make idempotency necessary
Partitioning and Consistent Hashing
Splitting data across machines without reshuffling everything when the machine count changes, plus how location-based data is partitioned
Distributed Primitives
Leader election, consensus, distributed locks and logical clocks — the building blocks that make many machines behave like one