Lesson 18: Capstone — Design & Run a Real System

Seventeen lessons, each solving one problem in isolation. This lesson puts them together: take a spec, estimate, pick the minimal architecture, run it for real, measure, find the bottleneck, fix it, then measure again. The most valuable part isn't the final architecture — it's the before/after table for every step, because that's what shows which step was worth it and which one just added complexity.

The overall result: from 6,115 up to 25,786 requests/second (4.2 times) and p99 from 14.71 down to 1.28 ms (11.5 times). But two more memorable numbers sit in the middle. Caching — the optimization everyone reaches for first — eliminates 99% of read queries but only lifts throughput 16%, because the real bottleneck is on the write path. And a read replica, a perfectly reasonable optimization on paper, delivers exactly 0% improvement — actually slightly worse.

18.1 Take the spec, and pin down the requirements

The spec: a URL shortener with click statistics. This spec is chosen deliberately, because it packs three very different kinds of load into a system small enough to read in one sitting:

The last column decides the entire design, and it only surfaces if you're willing to ask. Three paths, three different consistency requirements, so they deserve three different treatments — and as the measurements will show, that's exactly where most of the performance lives.

Before drawing anything, here are the questions that must be asked. Without answers, every choice after this is a guess:

• DAU and peak RPS? Decides whether horizontal scaling is even needed.
• Read:write ratio? Decides whether caching and replicas are worth it.
• How long do links live? Decides storage capacity and cleanup strategy.
• How accurate do the stats need to be? Decides whether async is allowed.
• What's the SLO? Decides when "good enough" is reached and optimizing should stop.

18.2 Estimate and design v1

Say the numbers you get back are 10 million redirects a day and 100,000 new links a day. Back-of-envelope, the same way Lesson 1 did it:

The last two rows already tell us something important before a single line of code gets written: the click table is 25 times bigger than the link table. The real problem here isn't storing links, it's handling clicks — and that's the first hint that the analytics write path deserves different treatment.

What about that 350 req/s peak? It's tiny. A single server handles it easily. So v1 is deliberately minimal: one service, one PostgreSQL, no cache, no queue. Click writes go straight into the clicks table right inside the redirect request.

18.3 Build it for real, and measure the baseline

v1 runs. Now the most important step, and the one most often skipped: measure before optimizing. Without a baseline, every improvement after this is just a feeling, and you have no way to know whether a change was worth keeping.

$ ./tools/capstone-test.sh v1
  v1  rps=   6115.1  p50=  2.91  p95=  4.93  p99=  14.71
      dbReads=  90431  dbWrites=  90431  redisOps=       0  hitRatio=None

Those two lines already contain the whole clue. The number of read queries and the number of write commands are exactly equal: 90,431 and 90,431. That means every redirect generates one write to the database — and a write is far more expensive than a read, and it sits right on the hottest path in the system.

The common instinct here says: "read-heavy, so add a cache." That instinct isn't wrong, but it answers the wrong question. Let's follow it first anyway — because that's exactly what everyone would do — and then see what the data says.

18.4 The measure → patch → measure loop

Four rounds, each changing exactly one parameter. All of it real measurements on the same machine, the same load scenario:

v1 → v2: caching eliminates 99% of read queries but only lifts throughput 16%. If you only looked at the "read queries" column, this looks like a huge win — 90,431 down to 1,006. But the throughput column tells a different story, and the reason is in the last column: the number of writes doesn't drop at all, and even rises with traffic. Caching fixed something that wasn't the bottleneck. The one bright spot is p99, down 2.5 times (14.71 → 5.94), because reads no longer have to queue for the connection pool alongside writes.

v2 → v3: changing how clicks are counted, throughput 3.6 times. This is the most worthwhile step, and it's not a technical trick, it's a requirements decision: accepting that statistics are allowed to lag by a few seconds. Recognizing that back in section 18.1 is what makes this step possible. After v3, the system's hot path never touches the database at all — 0 reads, 0 writes.

v4 → v5: rate limiting is nearly free. p50 goes up 0.06 ms — exactly one network round trip to Redis, matching the number from Lesson 13. And just as in Lesson 13, p95/p99 actually measure lower (1.01 and 1.28 versus 1.49 and 1.85). This effect repeats reliably across two different lessons but the mechanism hasn't been proven, so it's recorded here as an observation, not a conclusion.

What about the cost of v3? It's real, it just doesn't show up in the throughput column. The /stats endpoint exposes it:

$ curl -s localhost:3010/stats/k7
{
  "code": "k7",
  "clickTrongDb": 191,          // already flushed to the database
  "clickConTrongBoDem": 1028,   // still sitting in Redis, not flushed yet
  "tong": 1219                  // the CORRECT number only exists when you add both
}

The gap between 191 and 1,219 is eventual consistency in its most concrete form (Lesson 9). Anyone reading the clicks table directly in the database sees a wrong number. That's the price paid for 3.6 times throughput — and for click statistics that's the right price to pay; for an account balance, it isn't.

18.5 Summing up the trade-offs, and what this lab can't tell you

Seventeen lessons, and each one boils down to one trade-off sentence:

Looking at the whole table at once reveals the most important point: no row has an empty "cost" column. Nothing in this series is free. The right question was never "should I use X" — it was always "is the problem I have worth X's price" — and that question can only be answered with measurements from your own system.

Self-assessment checklist for a design

• Do you have numbers yet — peak RPS, read:write ratio, storage, SLO?
• Has the minimal version measured a baseline, or are you optimizing in the dark?
• Does every added component have before/after numbers proving it was worth it?
• Does every network call have a timeout, and do timeouts shrink with depth?
• Which operations can be retried — are they idempotent yet?
• How many tiers retry? (The right answer is almost always: one.)
• Under overload, what do you drop first, and has that been decided already?
• Is there a correlation ID all the way through, including across queues?
• Does the fallback path depend on fewer things than the primary path?
• When was the failover mechanism last drilled?

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab

./tools/capstone-test.sh seed   # seed 1,000 links (run once)
./tools/capstone-test.sh all    # run v1 -> v5 in sequence

#   v1 baseline             rps  6,115  p99 14.71ms   dbReads 90,431  dbWrites 90,431
#   v2 + cache              rps  7,122  p99  5.94ms   dbReads  1,006  dbWrites 103,775
#   v3 + async click counts rps 25,853  p99  1.75ms   dbReads      0  dbWrites      0
#   v4 + read replica       rps 24,803  p99  1.85ms   (!) NO improvement at all
#   v5 + rate limit         rps 25,786  p99  1.28ms

# The visible cost of v3:
curl -s localhost:3010/stats/k7   # inDb 191 · stillBuffered 1028 · total 1219

In summary

Process matters more than architecture. Ask for the numbers before drawing anything; design the minimal version; measure the baseline; then each round changes exactly one thing, with before/after numbers recorded. It sounds slow, but it's the only way to know whether you're actually improving things or just adding complexity.

Three numbers from this capstone are worth carrying with you. Caching — the optimization everyone reaches for first — eliminates 99% of read queries but only lifts throughput 16%, because the bottleneck sits somewhere else. What's genuinely worthwhile is a requirements decision, not a technical trick: accepting that click statistics can lag by a few seconds, and throughput goes up 3.6 times. And a read replica, reasonable on every diagram, delivers exactly 0% — because the cache had already absorbed all the reads it was meant to carry.

And the last point, also the single most memorable thing across all 18 lessons: the trade-off table in section 18.5 has no row with an empty "cost" column. Every technique in this series has a price. A designer's job isn't knowing lots of techniques — it's knowing when that price is worth paying — and the answer always lives in the data of your own system, not in any article, including this one.

This is the last lesson of the roadmap. If you've run every lab and measured it yourself on your own machine, you don't just know these concepts — you've watched them happen. That's the difference this whole series was written to create.