Two parallel tracks: see it, then run it
Most system design material teaches through static diagrams and sentences: "add a cache to reduce load", "use a load balancer to scale out". The trouble is that the things that matter most in a distributed system β queueing delay, replication lag, cascade failure, the cost of one network hop β are all dynamic behaviour. A diagram can only hint at them; to understand you have to watch it happen, and to believe it you have to measure it yourself.
So every lesson in this series runs on two parallel tracks:
- The "see it" track β simulation right on the page. Drag the RPS slider and watch p99 go vertical, kill a replica and watch the queue pile onto the survivor, enable retries without jitter and watch the system knock itself over. Nothing to install.
- The "run it" track β a Docker lab on your machine. Real nginx, a real Node app, real Redis and PostgreSQL; you generate the load and read your own machine's numbers. Every figure in these lessons comes from here, and none of them is invented.
/fast endpoint fell from around 29,000 req/s to 116β137 req/s and p99
jumped from 0.71 ms to around 160 ms β a collapse of 200β250Γ
(repeated 3 times). That number changes how you write code in a way a sentence never does.
About this series
The backbone of the simulation track is a hand-written discrete-event simulation core
(sysdesign-sim-engine.js) β checked against M/M/1 queueing theory and Little's Law, with
an error under 4% at every load level. You can verify it yourself with
node sysdesign-engine-selftest.mjs (42 assertions). The centrepiece is the
Traffic Lab: build a topology, drag the load up, inject failures, and watch the
system degrade or collapse outright.
Open the Traffic Lab β
Before you begin
Recommended: the SQL series β indexes, query plans and transactions/ACID all return in Lessons 7β9; the Git series for the deployment workflow in Lessons 15β17.
For the "run it" track: Docker and Docker Compose v2. Without Docker you can still do the entire simulation track β but you lose the most valuable part, measuring real numbers yourself.
linux/amd64, the container runs
through an emulation layer: many times slower, with heavily noisy latency. Docker prints
one faint warning line and carries on regardless, so it is very easy to miss β and from then on
every measurement you take is meaningless. Lesson 2 includes a mandatory preflight
check for exactly this.
Foundational glossary
These terms recur across all 18 lessons, so learning them up front makes the reading much lighter:
| Term | What it means in this series |
|---|---|
| Latency | The time to serve one request. Distinct from throughput β and optimising one usually harms the other. |
| Throughput | The number of requests served per unit of time (req/s). |
| p50 / p95 / p99 | Latency percentiles. p99 = 99% of requests are faster than this value. The mean hides the tail, so p99 is what reflects the worst experience users actually get. |
| Utilisation (Ο) | How occupied a resource is, Ο = Ξ»/ΞΌ. As Ο approaches 1, latency approaches infinity β not linearly. |
| Little's Law | L = λ·W. The number of requests inside the system equals the arrival rate times the time each request spends inside. Used to size worker counts and connection pools. |
| Backpressure | Signalling upstream that the system is overloaded, instead of silently accepting more and queueing without bound. |
| Cache hit ratio | The share of requests answered by the cache. Raising it from 90% to 99% reduces load behind it by 10Γ, not by 10%. |
| Replication lag | How far behind the primary a replica is. The user-visible consequence: write something, read it back, get the old data. |
| Consistent hashing |
Assigning keys to nodes such that adding or removing one node migrates only about 1/N of the keys,
rather than nearly all of them as hash % N does.
|
| Idempotency | The property that repeating an operation does not change the result. It is the precondition for safe retries, and the only way to produce an exactly-once effect. |
| Cascade failure | A failure that propagates: one slow dependency drains the caller's connection pool, which spreads back up through the whole system β usually amplified by retries. |
| SLI / SLO / error budget | A metric measured from the user's side / a target with a number attached / the allowance of failure you may spend. Used to regulate how fast you ship features. |
The 18-lesson roadmap
The order follows the dependency chain: each lesson uses only concepts taught in the lessons before it. A few links are placed deliberately β idempotency before message queues (because at-least-once delivery inevitably produces duplicates), and microservices near the end (you can only judge the trade-off properly after paying for network hops, distributed locks and eventual consistency yourself). Lessons unlock progressively as they are published.
Latency, Throughput & Queueing Theory
Tail latency and why p99 is the number that counts; latency orders of magnitude from L1 cache to an intercontinental round trip; Little's Law; why at Ο = 0.7 the queueing time is already 2.3 times the service time. Demo: the Traffic Lab on a single node, with the theoretical curve drawn over the simulated numbers.
Building the Lab & Measuring One Server's Limits
The life of a request through the kernel queue β libuv β the event loop; blocking handlers and what they really cost; measuring properly (warm-up, coordinated omission, do not fight the server for CPU); finding the knee point. Lab: standing up the Docker stack used by the whole series.
Scaling Out & Load Balancing
L4 versus L7; round robin / least connections / random of two choices; active versus passive health checks; measurement shows graceful shutdown changes nothing for short requests but cuts the latency tail 2.9 times for long ones; what sticky sessions cost.
Reverse Proxy & API Gateway
Three terms people use interchangeably are three levels of responsibility; TLS costs about 10% with
keep-alive but nearly 16 times without it; why you must never trust a client-supplied
X-Forwarded-For; the static-file measurement that contradicts the prediction; parallel
aggregation at 208 ms against 386 ms sequential; the gateway as a single point of failure.
Caching: Cache-Aside, TTL & Invalidation
Every extra nine in your hit ratio divides database load by ten; measurement shows a 4-point drop in hit ratio makes p99 jump 9.4 times while p50 does not move; four caching patterns; TTL versus explicit deletion versus versioned keys; single-flight cuts database queries 20-fold but makes p50 worse; a badly designed cache key drives the hit ratio to zero.
CDN & Edge Caching
The speed of light is a hard limit no code gets past; the edge tier measures 25 times the throughput
with the origin handling just 4 of 208,173 requests; no-cache still caches;
ETag and 304; serving stale when the origin is down β and the limits of
that; a cache key containing tracking parameters raises origin load 251 times while every
client-side metric stays identical.
Replication & Scaling the Read Tier
The three purposes of a replica need three different configurations; going from async to sync raises
write latency 79% and cuts write throughput 44%; measurement shows a lag of just 0.557 ms still
breaks read-your-writes on 87.21% of reads; both fixes reach 0% at a cost of 4.6% and 12.6%
throughput; split-brain reproduced for real with pg_promote.
Sharding & Consistent Hashing
Why sharding is the last step; modulo hashing from 4 to 5 shards forces 79.97% of the data to migrate while consistent hashing moves only 18.78%; without virtual nodes one shard receives 66 keys and another 25,465; a skewed shard key gives better p50 and p95 but a p99 that is 10.3 times worse.
CAP & Consistency Models
P is not a choice; PACELC is the part you meet daily; the consistency spectrum with its measured
price; R+W>N gives exactly 0 stale reads while R+W=N still gives 33%;
last-write-wins loses data silently when clocks drift by 50 ms; a real partition leaves the first
request hanging for 30 seconds.
Distributed Locks
A lock for efficiency is not a lock for correctness; measurement shows a textbook-correct lock whose TTL is shorter than the work produces 79 conflicts out of 80 β worse than no lock at all; a GC pause longer than the TTL produces 31 out of 80 while Redis does nothing wrong; a fencing token prevents the damage, not the conflict.
Idempotency & Safe Retries
Exactly-once is an illusion β the two generals problem; an idempotency key identifies the intent,
not the attempt; deduplication and business logic must share one transaction; measurement of 78,774
concurrent requests on one key yields exactly 1 record and 1 201, and with it switched
off, 14,236 records and a negative balance.
Message Queues & Asynchronous Processing
Acknowledging at the wrong moment loses 352 jobs without a trace β the maximum number lost is exactly the prefetch size; a poison message does not block the queue in Redis Streams, it accumulates silently; four times the consumers is only 2.68 times faster; queue depth is the number one health metric.
Rate Limiting & Backpressure
A fixed window lets 200 requests per second through when the limit is 100 β
measured right at the window boundary; the four algorithms barely differ in speed but differ more
than tenfold in memory; INCR and EXPIRE issued separately leave 20 of 200
keys stuck forever; three replicas sharing one Lua script hold the global limit to within 0.13%.
Event Sourcing & CQRS
A non-idempotent projection replayed a second time reports a balance exactly 2.00 times too high with no warning anywhere; replaying 200,000 events takes 235 ms, so fixing a bug and rebuilding is an office-hours job; snapshots are 6.6 times faster on long aggregates but slower on short ones.
Monolith vs Microservices
The same four-step use case: the monolith at p99 2.06 ms and 18,887 req/s, the microservices at 8.25 ms and 3,744 req/s; compounding availability matches the formula to within 0.1 percentage points; a saga missing its compensating action leaves 200 paid orders and 0 stock reservations.
Observability β Metrics, Logs & Tracing
For an incident making 1% of requests 300 ms slower, the measured p99 is 0.90 ms β
completely blind; only p99.9 reveals 474 ms; adding a single user_id label takes the
time-series count from 2 to 49,317; forgetting to propagate a correlation ID does
not leave you short of data, it leaves you with WRONG data β 100% of the time attributed to an
innocent service.
Failure Modes & Resilience
A three-tier chain retrying three times at each tier turns 30 requests into 810 hits on the service at the bottom; a retry budget brings that down to 69; backoff without jitter clusters 19.6 times worse than with jitter, which manages 5.3 times and still finishes sooner; a timeout in the wrong place has the server complete 50,000 ms of wasted work while the dashboard stays green.
Capstone β Design & Run a Real System
All 17 lessons combined into a working URL shortener: throughput Γ4.2 and p99 11.5 times better. But caching β the optimisation everyone reaches for first β removes 99% of read queries and adds only 16% throughput; and a read replica, sensible on every diagram, delivers exactly 0%.
Related series
This series overlaps with several others on the blog, and they can be read across: SQL in the Browser (indexes, query plans, transactions/ACID β the foundation for Lessons 7β9), Vector Databases (indexing and clustering, and the sharding problem itself), AI Systems Engineering (orchestration, blackboard, deadlock β the same family of problems as Lessons 15β17).
Comments