Lesson 9: CAP & Consistency Models

Lesson 7 section 7.4 left a question open: when the network is cut, neither node knows whether the other has died or has merely lost contact — from the inside, those two situations are completely identical. This lesson formalises that question, and more importantly, points out the part CAP leaves out but that you meet every day.

This lesson's lab creates a real partition with docker network disconnect, and the answer turns out to be neither "an error" nor "stale data" but three different things across three consecutive calls: the first request hangs for 30 seconds, the next two error out in 29–40 ms, and writes to the primary keep succeeding normally. The consistency model you think you have and what the system actually does when the network breaks are usually two different things.

9.1 Reading CAP correctly: P is not a choice

CAP is usually told as "pick two of three: Consistency, Availability, Partition tolerance". That telling leads to a wrong conclusion from the very start, because P is not something you pick. The network will be cut: a cable is severed, a switch dies, a region loses connectivity, or a container simply leaves the network. That is physical reality, not a line in a config file.

The correct statement: while a partition is happening, you must choose between C and A. Outside of that, the question does not exist. Which is why the notion of "a CA system" is a confusion — it only means "a system that has never met a partition", which is to say a system that has not run long enough.

Lab: a real partition, and three different behaviours

Theory says "choose C or A". But when a partition really happens, what does your system do? The answer is usually not what you think. Cut the replica off the network while the app is reading from it:

cd blog/sysdesign/sysdesign-lab
docker compose --profile replica up -d
NET=sysdesign-lab_default

# 1) Normal: reading from the replica works
curl -s "http://localhost:3001/rww?id=7"
# {"wroteVersion":1,"readVersion":1,"readFrom":"replica","stale":false}

# 2) CUT the replica off the network — a REAL partition at the network layer
docker network disconnect $NET sysdesign-lab-postgres-replica-1

# 3) What does the app do?
curl -s -m 30 "http://localhost:3001/rww?id=7"   # HANGS 30 SECONDS, client gives up
curl -s -m 30 "http://localhost:3001/rww?id=7"   # 40ms: ENOTFOUND postgres-replica
curl -s -m 30 "http://localhost:3001/rww?id=7"   # 29ms: ENOTFOUND postgres-replica

# 4) But what about WRITES to the primary? The primary was not cut off.
docker compose exec -T postgres psql -U lab -d lab -tAc \
  "UPDATE profiles SET version=version+1 WHERE id=7 RETURNING version;"
# 6 · UPDATE 1 · took 135ms — STILL SUCCEEDS

# 5) Reconnect. --alias is MANDATORY, otherwise DNS never comes back.
docker network connect --alias postgres-replica $NET sysdesign-lab-postgres-replica-1
curl -s "http://localhost:3001/rww?id=7"   # {"readVersion":9,"stale":false}
curl -s "http://localhost:3001/rww?id=7"   # {"readVersion":9,"stale":true} — catching up

9.2 PACELC — the part CAP leaves out, and the part you meet daily

CAP only speaks about the moment of a partition. But partitions are rare; the other 99.9% of the time the network is perfectly fine — and throughout that time there is still a trade-off, CAP just does not mention it. PACELC adds exactly that part:

if (P) then (A or C) else (L or C) — if there is a partition, choose between availability and consistency; otherwise (Else), choose between latency and consistency.

The second half is the half you meet every day, and you already met it in Lesson 7 without naming it: reading from a replica is fast but may be stale; reading from the primary is fresh but slower and consumes the primary's resources. That is exactly "L or C" — the network is not cut at all, every node is healthy, and you still have to choose.

9.3 The spectrum of consistency models

"Consistency" is not an on/off switch but a range. Each stronger level eliminates one kind of anomaly the user can perceive — and its price is latency or availability.

9.4 Quorums: why $R + W > N$

With $N$ replicas, write to $W$ of them and read from $R$ of them. If $R + W > N$ then by the pigeonhole principle the set of $W$ nodes just written and the set of $R$ nodes being read must share at least one node — and that shared node holds the newest version. That is the whole idea.

What read-repair does — and does not do

Read-repair is this mechanism: when a read finds that some node in the $R$ set holds an old version, it writes the new version across to that node there and then. The interesting part is that it does not reduce the stale-read rate of that very read — it makes the divergence converge.

Write once to $W=1$ node, then read 40 times in a row with $R=2$, $N=3$ (20,000 trials):

The two rows start out identical — 33.7% — and that is the main point: read-repair does nothing for the first read. But without it, the rate stays at 33% forever. That is the concrete meaning of an "eventual consistency" system that is never eventual: if no mechanism propagates the new version to lagging nodes, they stay stale permanently.

9.5 Linearizability, serializability, and how LWW loses data

These two words get used interchangeably but describe two different things, and knowing the difference helps you ask the right question when reading a database's documentation:

The practical consequence: a database can be serializable without being linearizable (the transactions are correct but you are reading from a stale replica), and vice versa. When documentation says "we support serializable", that is a statement about transactions, not a promise that your reads will see the freshest data.

Last-write-wins: losing data without a sound

When two writes arrive together, there has to be a rule for who wins. The simplest rule — and the most common — is last-write-wins: the version with the larger timestamp wins. The problem: "later" is decided by a wall clock, and two machines' clocks never agree exactly.

The scenario: client A writes at $t=1000$, client B writes 50 ms later at $t=1050$. But B's clock is skewed. Run for real:

The threshold is exactly the time gap between the two writes: a clock skew of just 50 ms — precisely the gap between them — is enough for the later write to be dropped. And it is dropped silently: both clients receive a success response, there is no error, and no log records that a value was overwritten by an older one.

function lastWriteWins({ clockSkewMs }) {
  // The REAL moment (by an imaginary perfect clock), in ms.
  const realTimeA = 1000;
  const realTimeB = 1050; // B writes exactly 50 ms AFTER A

  // But each client stamps using ITS OWN clock.
  const stampA = realTimeA + 0;
  const stampB = realTimeB + clockSkewMs; // B's clock is skewed

  const writes = [
    { client: 'A', value: 'balance = 100', realTime: realTimeA, stamp: stampA },
    { client: 'B', value: 'balance = 150', realTime: realTimeB, stamp: stampB },
  ];

  // LWW: the version with the HIGHEST timestamp wins.
  const winner = writes.reduce((a, b) => (b.stamp > a.stamp ? b : a));
  // What causality actually requires: the write that happened LATER in real time wins.
  const shouldWin = writes.reduce((a, b) => (b.realTime > a.realTime ? b : a));

  return {
    clockSkewMs,
    stampA,
    stampB,
    lwwWinner: winner.client,
    correctWinner: shouldWin.client,
    lostWrite: winner.client !== shouldWin.client,
    lostValue: winner.client !== shouldWin.client ? shouldWin.value : null,
  };
}

There is one more thing the table above implies: everything here depends on how far the clocks are skewed — a quantity you do not control and usually do not measure. Lesson 10 continues exactly along that line: a distributed lock also rests on an assumption about time, and when that assumption is wrong two workers both believe they are holding the lock — while Redis behaves perfectly correctly throughout.

Reproduce the measurements yourself

# Sections 9.4 and 9.5 — no Docker needed, runs in a few seconds
cd blog/sysdesign
node sysdesign-quorum.js

# Section 9.1 — a REAL partition between the app and the replica
cd sysdesign-lab
docker compose --profile replica up -d
NET=sysdesign-lab_default

curl -s "http://localhost:3001/rww?id=7"                         # normal
docker network disconnect $NET sysdesign-lab-postgres-replica-1  # CUT THE NETWORK

time curl -s -m 30 "http://localhost:3001/rww?id=7"  # hangs 30s — NO timeout in the app
curl -s -m 30 "http://localhost:3001/rww?id=7"       # ENOTFOUND in ~40ms

docker compose exec -T postgres psql -U lab -d lab -tAc \
  "UPDATE profiles SET version=version+1 WHERE id=7 RETURNING version;"   # WRITE still OK

# --alias is MANDATORY: without it DNS never comes back and the lab stays broken
docker network connect --alias postgres-replica $NET sysdesign-lab-postgres-replica-1
curl -s "http://localhost:3001/rww?id=7"

In summary

P is not a choice — the network will be cut. The choice between C and A only appears during a partition, and it is a choice per operation, not for the whole system. The part you meet daily is the second half of PACELC: when the network is perfectly healthy, every read path still has to choose between latency and freshness — and if you do not choose, latency has been chosen for you.

$R + W > N$ guarantees the read set and write set overlap, measured at exactly 0 stale reads across 100,000 rounds; but $R + W = N$ still gives 33%, so the inequality has to be strict. A quorum gives you overlap, not linearizability — that needs a consensus protocol. And read-repair does not improve the first read; it is what turns a 33% that stays put forever into 33% → 11.2% → 0.3% → 0%.

The most memorable finding of this lesson came from the lab: a partition does not show up as an error but as silence — the first request hung for 30 seconds, no error, no connection close. Which means that if you set no timeout you have chosen A by accident, and that is the worst of the three options: neither consistent nor available.