Lesson 12: Message Queues & Asynchronous Processing

Lesson 11 concluded that at-least-once plus idempotent processing is the only workable combination. A message queue is where that combination stops being a choice and becomes a requirement: when a job can be redelivered at any moment, the consumer must be idempotent — otherwise every redelivery is another side effect.

This lesson's lab stands up a real producer/consumer on Redis Streams. Two measurements stand out. First, going from 1 consumer to 4 is only 2.68 times faster, not 4 — because the bottleneck has moved from the consumers to Redis itself. Second, acking at the wrong moment makes 352 jobs vanish without a trace: the number of recoverable jobs is 0 instead of 352.

12.1 Why go asynchronous

12.2 Ack semantics — and 352 vanished jobs

The entire reliability of a message queue rests on when you send the ack. An ack means "this job is done, stop redelivering it" — so sending it too early is throwing away your own ability to recover.

The mechanism behind that has a name: when the broker hands out a job it does not delete it but moves it to the pending list and starts a timer. If the ack arrives before the timer runs out, the job counts as done. If it does not — the worker died, or is simply too slow — the broker concludes the worker is broken and hands the job to somebody else. That waiting period is called the visibility timeout, and it is exactly what turns "the worker died" from data loss into a redelivery.

The lab measures the damage. The consumer reads 500 jobs ahead at a time (prefetch 500) and each job takes 5 ms — so one batch takes about 2.5 seconds. We docker kill the consumer after 1.2 seconds, which is mid-batch:

// BLOCK 1000: wait up to 1s if the stream is empty, instead of spinning and burning CPU.
const res = await redis.cmd('XREADGROUP', 'GROUP', GROUP, CONSUMER,
  'COUNT', String(BATCH), 'BLOCK', '1000', 'STREAMS', STREAM, '>');

for (const [id, fields] of entries) {
  const f = {};
  for (let i = 0; i < fields.length; i += 2) f[fields[i]] = fields[i + 1];

  // Dedup on event_id BEFORE causing any side effect. SET NX is atomic, so if two
  // workers receive the same event only one wins (the precondition of at-least-once).
  const fresh = await redis.cmd(
    'SET', `lab:q:seen:${f.event_id}`, '1', 'NX', 'EX', '3600');
  if (fresh !== 'OK') {
    await redis.cmd('XACK', STREAM, GROUP, id);   // already processed -> ack and skip
    continue;
  }

  await doWork(f);

  // ACK LAST. This single line decides the reliability of the whole system.
  await redis.cmd('XACK', STREAM, GROUP, id);
}

12.3 Poison messages and DLQs — and one thing usually said wrongly

A poison message is a job that can never be processed: a malformed payload, a reference to a deleted record, or one that hits a bug exactly. It fails, is redelivered, fails again — forever.

The standard treatment is a DLQ — short for dead letter queue. It is simply a second queue: once a job has failed more times than allowed, instead of letting it come back forever, you move it there and ack it on the main queue. The job is not lost, but it no longer disturbs the normal flow — it sits waiting for a human to look at it.

The lab pushes 5,000 jobs with one poison job every 500 (10 poison jobs in total), and one consumer:

12.4 Ordering and consumer groups — adding a consumer breaks the order

First, the name in the heading. A consumer group is the mechanism that lets several consumers read one queue without treading on each other: the broker remembers which consumer each job went to, and each job is handed to exactly one member of the group. That is why the command in section 12.2 is XREADGROUP and not XREAD — adding a consumer to the group adds processing capacity rather than duplicating the work. The trade-off is what this section is about.

Ordering is only guaranteed within one partition (or one stream key). The moment you add a second consumer to scale, jobs are processed in parallel and there is no global order left.

The lab's scaling measurements show exactly that — plus something nobody expects:

Four times the consumers is only 2.68 times faster. And the "rate per consumer" column explains why: it falls from 11,338 to about 7,600. The bottleneck has moved from the consumers to Redis — Redis executes commands on a single thread, so four consumers are contending for one sequential resource.

12.5 Backpressure and queue depth

Queue depth is the number one health metric of an asynchronous system, and the reason lies back in Lesson 1: if the arrival rate $\lambda$ exceeds the service rate $\mu$ then depth grows without bound, and by Little's Law so does the waiting time. No load level is "safe" if $\lambda > \mu$ — there is only the time before the consequences show.

Four relief valves, in the order to try them: scale the consumers (if the bottleneck really is there — see section 12.4); cap the queue length and reject new jobs beyond it, so you fail fast rather than promise; load shedding — drop unimportant jobs to protect important ones; and slow the producer down (rate limiting, Lesson 13). What all four have in common: they are all ways of saying no — and an asynchronous system with no way to say no is merely postponing saying it.

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab
docker compose --profile queue up -d redis

# ./tools/queue-test.sh COUNT N_CONSUMER [WORK_MS] [POISON_EVERY] [MAX_ATTEMPTS] \
#                       [IDEMPOTENT] [ACK_MODE] [DUR] [BATCH]

# --- Scaling: 20,000 jobs, 1 / 2 / 4 consumers. Read `msActive`, not `rate`. ---
./tools/queue-test.sh 20000 1 0 0 0 0 after 15000   # msActive 1764ms · 11,338/s
./tools/queue-test.sh 20000 2 0 0 0 0 after 15000   # msActive ~1201ms · total 16,652/s
./tools/queue-test.sh 20000 4 0 0 0 0 after 15000   # msActive ~658ms  · total 30,375/s

# --- Poison messages: 5,000 jobs, one poison job every 500 ---
./tools/queue-test.sh 5000 1 0 500 0 0 after 15000  # pending=10, stuck forever
./tools/queue-test.sh 5000 1 0 500 1 0 after 15000  # dlq=10 · pending=0

# --- Acking at the wrong moment: kill the consumer MID-BATCH of 500 jobs ---
docker compose exec -T redis redis-cli DEL lab:jobs
docker compose run --rm --no-deps -e ROLE=producer -e COUNT=3000 queueworker queue.js
docker compose run --rm --name qkill --no-deps -e ROLE=consumer -e WORK_MS=5 -e BATCH=500 \
  -e ACK_MODE=on-receive -e DURATION_MS=30000 queueworker queue.js &
sleep 1.2 && docker kill qkill
docker compose exec -T redis redis-cli XPENDING lab:jobs g1 | head -1
# prints 0  -> the 352 unprocessed jobs are gone for good
# Switch ACK_MODE to `after` and repeat: it prints 352, all recoverable

In summary

Going asynchronous decouples user latency from heavy work and absorbs bursts, but it changes the shape of the problem rather than removing it: the peak becomes a backlog, and backlogs have their own price.

All the reliability rests on when you ack. Measured with a prefetch of 500 and the worker killed mid-batch: ack-on-receipt gives 0 recoverable jobs, ack-after-completion gives 352. The maximum number of jobs lost equals the prefetch size exactly — so prefetch is a reliability parameter, not just a performance one. And because at-least-once means a job may be processed twice, the consumer must be idempotent (Lesson 11).

On poison messages, I corrected a claim that is usually stated wrongly: with Redis Streams a poison message does not block the messages behind it — the other 4,990 jobs ran normally, and the broken message merely stayed in the pending list. That makes the damage quieter, so for this kind of broker the mandatory thing is an alert on pending-list length, not just a DLQ.

Finally, four times the consumers is only 2.68 times faster — the fourth time in this series that the bottleneck moved to a shared component the moment we scaled out. The cheapest check is the rate per worker: if it falls as you add workers, you are re-slicing a fixed cake.

Lesson 13 covers the last relief valve on the list in section 12.5: rate limiting. It is the mechanism by which a system says no in a controlled way — instead of accepting everything and breaking, or accepting everything and promising a six-hour backlog.