Lesson 10: Distributed Locks

Lesson 9 ended on the point that everything depends on how far the clocks are skewed. A distributed lock is the most direct example: it is a mechanism resting on an assumption about time, and when that assumption is wrong, two workers both believe they hold the lock — while Redis behaves entirely correctly and no command fails.

One term used throughout the lesson needs saying up front: a critical section is a stretch of code where only one worker may run at a time — for instance the part that reads a balance, computes something, and writes the new balance back. Inside a single-threaded JavaScript process you barely ever meet the concept, because nothing runs in parallel. But once there are two processes on two machines it returns — and the language can no longer protect it for you.

This lesson's lab has two real Node workers contend for one Redis lock and counts the conflicts: how often two or more workers were inside the critical section at once. One of the results made me re-run it several times to be sure: a lock that is implemented entirely correctly, but with a TTL shorter than the work time, produced 79 conflicts in 80 entries into the critical section — worse than using no lock at all (41/80). A broken lock is more dangerous than no lock, because it gives you confidence.

10.1 Two completely different needs

Before discussing how to implement a lock, you have to separate the two reasons for using one. Most material collapses them into one, and that is the origin of every distributed-lock accident.

10.2 The three mandatory parts of a Redis lock

A correct Redis lock needs exactly three things, and missing any one of them produces its own distinct bug:

/**
 * The three MANDATORY parts of a correct Redis lock (Lesson 10, section 10.2):
 *   NX      only set if the key does not exist — this is the "acquire" part
 *   PX ttl  a TTL is MANDATORY, otherwise a dead holder is a permanent deadlock
 *   token   a random value, so ONLY the owner can delete it (see release)
 */
async function acquire(key, token, ttlMs) {
  const res = await redis.cmd('SET', key, token, 'NX', 'PX', String(ttlMs));
  return res === 'OK';
}

/**
 * Releasing MUST be atomic: check the token and only then DEL, in one command.
 *
 * Why a plain `DEL` is wrong: your lock may already have EXPIRED and someone else may
 * have acquired it. A plain `DEL` would delete THEIR lock, and from there everything
 * breaks in a chain.
 */
const RELEASE_LUA = `
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end`;

async function release(key, token) {
  return redis.cmd('EVAL', RELEASE_LUA, '1', key, token);
}

Measured for real: does the lock do its job?

Two workers, each completing 40 critical sections (80 in total), with 300 ms of work each. The CONFLICT counter increases whenever two or more workers are inside the critical section at once:

Row B is what we expect: the lock works, 0 conflicts. Row C is the one worth stopping on for a long time. The lock in row C is implemented entirely correctly — it has NX, a TTL, a token, and a Lua release. There is no GC pause and no clock skew. It has exactly one wrong parameter: a TTL shorter than the work time. And the result is 99% — worse than using no lock.

10.3 Why Redlock is controversial

Redlock is an algorithm proposed to make distributed locking safer: acquire the lock on a majority of 5 independent Redis nodes. Martin Kleppmann argued against it, and the core of the argument is very compact — the problem is not Redis.

The lab experiment reproduces exactly that argument. The same correct lock, the same TTL of 1000 ms, longer than the 300 ms of work:

The mechanism: worker A takes the lock, then is stopped for 1200 ms — a stop-the-world GC, a migrated VM, a CPU-throttled container, or a machine that started swapping. Meanwhile the lock expires at the 1000 ms mark and worker B acquires it entirely legitimately. A wakes up at 1200 ms and still believes it holds the lock — it has no way to know that time has passed. It writes. Now two workers are writing.

10.4 Fencing tokens — the right answer for the correctness category

If a lock cannot guarantee "only one worker writes", the check has to move somewhere else: to the target resource itself. The lock issues a monotonically increasing number — a fencing token — and every write must carry it. The resource remembers the highest token it has ever seen and rejects anything lower.

That is how a "zombie" worker returning after a GC pause gets blocked — not because it knows it lost the lock, but because the resource knows somebody newer has been through.

// Fencing token: a MONOTONICALLY INCREASING number, issued by Redis via INCR (atomic).
// Taken AFTER acquiring the lock, so token order matches lock-acquisition order.
const fence = USE_FENCE ? Number(await redis.cmd('INCR', 'lab:fence:seq')) : 0;

/**
 * Write to the "resource", with fencing-token checking (Lesson 10, section 10.4).
 *
 * This is the crux: the target resource itself rejects any token LOWER than the highest
 * one it has seen. That is how a "zombie" worker returning from a GC pause — still
 * believing it holds the lock — gets blocked AT THE RESOURCE LAYER rather than by the
 * lock. If the resource does not check, the fencing token is just a decorative number.
 */
const FENCED_WRITE_LUA = `
local seen = tonumber(redis.call('GET', KEYS[1]) or '0')
local tok  = tonumber(ARGV[1])
if tok < seen then
  redis.call('INCR', KEYS[3])          -- count the REJECTED writes
  return 0
end
redis.call('SET', KEYS[1], tok)
redis.call('INCR', KEYS[2])            -- count the ACCEPTED writes
return 1`;

// In the worker: only do the work if the resource ACCEPTED the token.
let accepted = true;
if (USE_FENCE) accepted = (await fencedWrite(fence)) === 1;
if (accepted) await sleep(WORK_MS);

The same configuration that broke in experiment D — TTL 1000 ms, GC pause 1200 ms — but with fencing on:

10.5 Avoiding locks entirely — usually the best answer

After the four sections above, the pragmatic conclusion is: distributed locks are hard to get right, and even when correct they still do not provide the guarantee the "correctness" category needs. So in most cases the better question is not "how do I implement the lock" but "is there a way to not need one?"

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab
docker compose --profile lock up -d redis

# ./tools/lock-test.sh <LOCK> <FENCE> <PAUSE_MS> <TTL> <ROUNDS> <WORK_MS>
# The number that matters most is CONFLICTS: how often TWO OR MORE workers were
# inside the critical section at the same time.

./tools/lock-test.sh off 0 0    1000 40 300   # A: no lock         -> 41/80
./tools/lock-test.sh on  0 0    1000 40 300   # B: correct lock    ->  0/80
./tools/lock-test.sh on  0 0     200 40 300   # C: TTL < work time -> 79/80 (!)
./tools/lock-test.sh on  0 1200 1000 40 300   # D: GC pause > TTL  -> 31/80
./tools/lock-test.sh on  1 1200 1000 40 300   # E: + fencing -> 15 writes REJECTED

# A NOTE ON EXPERIMENT DESIGN: WORK_MS must be long enough relative to PAUSE_MS.
# Otherwise the other worker has already left the critical section by the time the
# "zombie" wakes up, and you measure 0 conflicts even though the lock really did fail.
# The first version of this lab made exactly that mistake.

In summary

You have to separate locks for efficiency (failure costs CPU) from locks for correctness (failure costs money). A distributed lock is only sufficient for the first, and the whole of section 10.3 is numerical evidence for that.

A correct Redis lock needs all three: NX, a TTL, and a random token with an atomic release. But being correct is still not enough — the strongest finding of this lesson is row C: a textbook-correct lock with a 200 ms TTL, shorter than the 300 ms of work, produced 79 conflicts out of 80, worse than using no lock at all (41/80). Because the lock synchronised the two workers into overlapping systematically instead of randomly.

With a GC pause longer than the TTL, conflicts were 31/80 — and not a single Redis command failed. That is why adding Redis nodes does not fix it: the wrong assumption is on the client side. Fencing tokens are the answer, but read them correctly: they do not prevent conflicts (still 15/80), they prevent damage — 15 writes rejected by the resource. And they only work if the resource genuinely checks the token.

Most pragmatic of all: avoid the lock. Atomic operations, optimistic concurrency, partitioning by key, transactions — none of them has to answer "what TTL is enough", a question that usually has no correct answer.

Lesson 11 goes into the strongest lock-avoidance option on that list: idempotency. If doing it twice gives the same result, there is no need to prevent the second time — and that is also the answer to a question every distributed system has to face: when a client receives a timeout, it cannot know whether the request was processed or not.