Lesson 14: Event Sourcing & CQRS

Every system in this series so far has stored the current state: the balance is 100, the avatar is avatar-v3.png, the order is in the shipped state. Every time something changes, we overwrite the old value. This lesson reverses that: store the sequence of things that happened, and treat the current state as merely something computed from that sequence.

The lab builds a real event store on PostgreSQL with 200,000 events. Three measurements stand out. First, a projection that is not idempotent, run a second time, produces a balance exactly 2.00 times too high — 9,132,892 instead of 4,566,446 — with no error thrown at all; the read model is simply wrong. Second, rebuilding the entire read model from 200,000 events takes 235 ms when replayed in application code, meaning replay at this scale is routine, not an emergency operation. Third, a snapshot makes a long aggregate 6.6 times faster but makes a short aggregate slower — it is a conditional tool, not a mandatory step.

14.1 State is the result of history

The same bank account, two ways to store it. The familiar way: one row, a balance = 100 column, every transaction is an UPDATE. The event-sourcing way: three rows, Deposited(+50), Deposited(+70), Withdrawn(-20), and 100 is the number we add up when we need it.

This bank account is an example of a concept that recurs throughout the lesson: an aggregate — the unit that an event history is grouped around, each with its own sequence of events and its own version number, entirely separate from every other unit. Your account is one aggregate; someone else's account is a different aggregate with its own event sequence. An order, a shopping cart, an insurance policy — each of those is an aggregate in exactly this sense too. By contrast, a row of system configuration or an entry in a product catalog usually does not need to become an aggregate this way: nobody needs to know who changed a product's description three months ago — see the pitfall right below.

The difference is not which way is "more correct" but what each preserves and what each loses. From a sequence of events, you can always compute the current state. From the current state, you cannot reconstruct the history — that information was overwritten by the UPDATE and no longer lives anywhere. This transformation is one-way, and event sourcing's entire value sits in the direction that gets lost.

Three abilities come nearly free from storing data this way. A full audit trail: every change is a row with a timestamp and a cause, not an updated_at column that just says "something changed". Debugging by rewinding: fold the log up to the exact moment right before the incident and look at the state then, instead of guessing from application logs. Answering questions nobody thought to ask at design time — and this is the most valuable ability: when the business side asks "how many people deposited money and then withdrew it all within 24 hours", with a state table you have to answer "we don't store that data"; with an event log, the answer is already sitting in the history.

14.2 Events are immutable facts

One very small naming convention decides almost everything that follows: events are named in the past tense. OrderPlaced, not PlaceOrder. The second one is a command — a request, which can be rejected, which can fail. The first one is a fact — it already happened, and there is no way to make it not have happened.

The last row is the one violated most often. When you discover an event recorded the wrong amount, the natural reflex is to UPDATE it to the correct value. Don't. Write a new AmountCorrected event — exactly the way an accountant never erases an entry but always writes a reversing entry. The reason is not purity but very practical: every projection already built from the old event still exists, and every projection built in the future will read the corrected one. The two worlds no longer agree, and nobody can trace why.

Immutability throws in one more thing for free: optimistic concurrency control. If every event carries a version number within its aggregate, and there is a UNIQUE(aggregate, version) constraint, then two processes that both read "the account is at version 1" and both try to write version 2 will produce exactly one winner. The lab runs exactly that scenario:

{
  "role": "concurrent",
  "ketQua": [
    { "who": "A", "ketQua": "THANG — su kien duoc ghi" },
    { "who": "B", "ketQua": "THUA — bi tu choi",
      "loi": "postgres: duplicate key value violates unique constraint \"events_aggregate_version_key\"" }
  ],
  "soSuKienCuoiCung": 2,
  "ghiChu": "Dung 2 su kien: version 1 va version 2. Nguoi thua KHONG de mat du lieu cua nguoi thang."
}

Worth comparing with an ordinary state table: two UPDATE balance = ... commands running at the same time mean the later one overwrites the earlier one, both report success, and nobody knows a change went missing. Here, the loser gets an error immediately — and "getting an error immediately" is something you can act on: reread the latest version, reapply the business rule, retry. This is exactly the lost update that Lesson 10 had to use a distributed lock to avoid; here it is blocked by a single UNIQUE constraint, with no lock at all.

14.3 Projections: read the log, build a read table

Nobody queries the event log directly to render a UI — folding up several thousand events on every page load is not viable. Instead, a projection worker reads the log and builds tables that are already optimized for each screen. A balances table is one projection. A recent-transactions list is another projection. A monthly report is a third projection. All three are built from the same log.

Lab measures both ways to build a read model from 200,000 events:

The number worth remembering is the second one, and it says this: at a scale of hundreds of thousands of events, rebuilding the entire read model is a matter of a few hundred milliseconds. At a scale of hundreds of millions, the same speed gives a few minutes. Both fall in the range of "runs during business hours", not "needs a maintenance window scheduled". That is why "fix the bug and replay" is a true statement rather than a slogan.

But it is only true when the projection is idempotent. The lab runs exactly one naive projection — the kind that simply accumulates every event it reads — and then runs it a second time:

// IDEMPOTENT=0 — projection ngay tho
{
  "soSuKienTrongLog": 200000,
  "dungPhaiLa": 4566446,
  "lan1": { "eventsApplied": 200000, "ms": 44, "tongSoDu": 4566446 },
  "lan2": { "eventsApplied": 200000, "ms": 39, "tongSoDu": 9132892 },
  "ketLuan": "SAI — read model lech 4566446 (gap 2.00 lan)"
}

// IDEMPOTENT=1 — co checkpoint `projection_state.last_seq`
{
  "lan1": { "eventsApplied": 200000, "ms": 43, "tongSoDu": 4566446 },
  "lan2": { "eventsApplied": 0,      "ms": 1,  "tongSoDu": 4566446 },
  "ketLuan": "DUNG — replay bao nhieu lan cung ra cung ket qua"
}

The most alarming detail is not the doubled number but the line "ms": 39: the wrong run succeeded, faster than the first run, with no warning at all. The read model is simply wrong from that second onward. The idempotent version applies 0 events on the second run and finishes in 1 ms — because it remembers which seq it already processed.

14.4 CQRS: splitting the write path from the read path

Once you have an event log and projections, the system naturally splits into two halves with two quite different sets of requirements. CQRS (Command Query Responsibility Segregation) is just the name for acknowledging that split and designing for it properly.

The last row is the one you pay for. The read model is built after the event has been written, so there is always a window where the log already has the event but the read table does not yet. That window is usually a few milliseconds and nobody notices — except exactly one person: the one who just clicked the button.

14.5 Running it for real: snapshots, schema change, and what you lose

Section 14.3 said replaying 200,000 events takes 235 ms. But that was replaying the whole system to build a read model — a rare event. The more frequent job is reading the current state of one aggregate to process a command, and once an aggregate accumulates tens of thousands of events, folding it from scratch every time is unacceptable. A snapshot is the answer: store the state at version N ahead of time, then fold only the events after it.

The second result is the more memorable one. For a short aggregate, a snapshot makes it slower — because it adds one more round of querying (read the snapshot, then read the tail), and the cost of that round is larger than what it saves. A snapshot is not a mandatory step of event sourcing; it is a conditional optimization, and the condition is that the aggregate has to be long enough. The right approach is to measure the distribution of aggregate length in your own system and set a threshold, instead of turning on snapshots for everything.

The second operational issue is that event schemas change over time. Last year's event had an amount field in dong; this year you add currency. The log is immutable so you cannot edit the old events — instead, the reading code has to understand both formats, usually through an upcasting layer: read an old-version event and lift it into the new shape right at load time. That layer only ever grows over the years and can never be deleted — it is a real debt, and it needs to be counted into the cost from the moment you decide.

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab

# --- Write 200,000 events into the append-only log ---
./tools/eventstore-test.sh seed 200000     # 1802ms · 111,003 events/s

# --- Run the projection AGAIN: idempotency decides everything ---
./tools/eventstore-test.sh replay2x 0      # naive:       run 2 -> 9,132,892 (DOUBLED)
./tools/eventstore-test.sh replay2x 1      # idempotent:  run 2 -> 0 events, 1ms

# --- Replay in application code: a more honest number than one GROUP BY ---
./tools/eventstore-test.sh projectApp      # 200k events / 235ms (174ms is pulling data)

# --- Snapshot only pays off for a LONG aggregate ---
./tools/eventstore-test.sh snapshot        # 100k events: 7.398ms -> 1.120ms  (6.6x)
./tools/eventstore-test.sh snapshot acc-0  # 200 events:  0.377ms -> 0.981ms  (SLOWER)

# --- Two people writing version 2 at once: UNIQUE(aggregate, version) blocks the lost update ---
./tools/eventstore-test.sh concurrent      # A wins · B gets a duplicate-key error

In summary

Event sourcing changes the question "what is the current state" into "what happened". That transformation is one-way: from the log you can always compute the state, from the state you cannot reconstruct the log. All the value — full audit trail, rewinding to debug, answering questions nobody thought to ask at design time — sits in the direction that gets lost.

The price is real too and needs saying in full: every engineer who touches that part has to learn one more model; the upcasting layer only ever grows over the years; and ad-hoc querying — the cheapest thing with a relational table — turns into "write a new projection, then wait for the replay". That is why event sourcing usually only earns its keep for a handful of core aggregates, not the whole system.

On operations, two numbers are worth carrying with you. Replaying 200,000 events in application code takes 235 ms — fast enough that "fix the bug and rebuild the read model" is a business-hours task, and that is this model's real superpower. But it is only true when the projection is idempotent: the naive version, run a second time, produces a balance exactly 2.00 times too high, successfully, faster than the first run, with no warning at all.

And a snapshot is not a mandatory step but a conditional optimization: 6.6 times faster with a 100,000-event aggregate, but slower by 2.6 times with a 200-event aggregate. It is a cache, so every rule of caching from Lesson 5 applies without exception — including the most annoying one: change the logic and you must wipe every snapshot at the same time you deploy.

Lesson 15 steps back to look at the whole picture: after paying for network hops, distributed locks, idempotency, and eventual consistency yourself, you finally have enough evidence to weigh the question every team runs into — whether to split a system into microservices, and what that split actually costs. This time the cost will be measured, not guessed at.