Lesson 2: Building the Lab & Measuring One Server's Limits

Lesson 1 closed on an unanswered question: what is $\mu$ — the maximum service rate of a server — in practice? That number cannot be read from documentation and cannot be deduced from the machine's specification. There is only one way: measure it.

This lesson stands up the measurement environment shared by all 18 lessons, then answers three questions with real figures: which queues a request passes through inside Node, how much damage one badly written handler does, and where your server saturates.

2.1 The life of an HTTP request in plain Node http

This lab uses Node's http module: no Express, no dependencies. The reason is not minimalism for its own sake — a framework hides precisely the places where the queues are, and queues are the main character of this whole series.

Before your JavaScript is called at all, a request has already passed through several layers — and each layer has a queue of its own:

2.2 The event loop and the blocking culprit

The lab's app has two endpoints that are equally slow in wall-clock terms but fundamentally different in nature — and that difference is the whole problem:

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

/** Block the event loop for `ms` — simulates a heavy synchronous handler. */
function blockFor(ms) {
  const until = Date.now() + ms;
  // Busy loop: does NOT yield to the event loop, so every other request queues up.
  while (Date.now() < until) {
    /* burning CPU on purpose */
  }
}

// ...inside the request handler:

    if (p === '/slow-async') {
      const ms = Number(url.searchParams.get('ms') || 50);
      await sleep(ms); // yields the event loop => other requests still get served
      return json(res, 200, { ok: true, mode: 'async', ms, instance: INSTANCE });
    }

    if (p === '/slow-sync') {
      const ms = Number(url.searchParams.get('ms') || 50);
      blockFor(ms); // BLOCKS the event loop => every other request waits
      return json(res, 200, { ok: true, mode: 'sync-blocking', ms, instance: INSTANCE });
    }

Called one at a time, these two endpoints return after the same interval. The difference only appears when other requests are in flight at the same time:

The real measurements

Background load is applied to /slow-sync?ms=20 with 4 connections while /fast is measured with 10 connections. Repeated three times, with nothing changed in between:

A 20 ms synchronous handler collapses a different endpoint's throughput by roughly 200–250×, and takes its p99 from 0.71 ms to about 160 ms. Not one line of /fast was changed — its only crime is sharing a process.

2.3 Standing up the lab shared by the whole series

The entire "run it" track across 18 lessons uses one Docker stack, extended gradually through profiles. You build it once here and then only switch on an extra profile when a later lesson needs it.

name: sysdesign-lab

# Shared config for every app server — a YAML anchor, so it is not repeated 3 times.
x-app-base: &app-base
  image: node:22-alpine
  working_dir: /app
  volumes:
    - ./app:/app:ro          # bind-mount: KHONG can Dockerfile, khong can npm install
  command: ['node', 'app.js']
  deploy:
    resources:
      limits:
        cpus: '1.0'          # CO CHU Y: xem phan cuoi bai
        memory: 256M

services:
  app1:
    <<: *app-base
    profiles: ['base', 'lb', 'cache', 'db']
    environment:
      INSTANCE: 'app1'
      DB_DELAY_MS: '40'
      GRACEFUL: '1'
    ports:
      - '3001:3000'          # chi app1 mo cong ra host, de bai nay goi truc tiep

  # ... app2, app3, lb, redis, postgres — enabled per profile ...

  loadgen:
    image: node:22-alpine
    profiles: ['tools']      # not started automatically; invoked by hand to measure
    working_dir: /loadgen
    volumes:
      - ./loadgen:/loadgen:ro
    entrypoint: ['node']
    deploy:
      resources:
        limits:
          cpus: '2.0'        # more than the app, so the tool is not the bottleneck

Bring it up and check the app is alive:

cd blog/sysdesign/sysdesign-lab
docker compose --profile base up -d

curl -s http://localhost:3001/health
# {"status":"ok","instance":"app1","uptimeSec":1,"inFlight":1,"totalRequests":2}

2.4 Measuring correctly

This section is why the lab uses a hand-written load generator (loadgen/loadgen.js) rather than wrk. Not because wrk is worse — it is considerably better — but because if the measuring tool is a black box, the lesson about how to measure disappears with it.

Four principles, and what ignoring them costs

1. Discard the warm-up. For the first few seconds the JIT has not optimised the code, caches are cold, and connections are not all open. Figures from this phase do not represent steady state. The lab's tool discards every sample collected before the warm-up mark, rather than averaging over the whole run.

2. Understand that the connection count is $L$. The tool runs closed-loop: each connection only sends its next request after receiving a response. So the connection count = the maximum number of concurrent requests = exactly the quantity $L$ in Little's Law from Lesson 1. This is not a trivial technicality — section 2.5 uses that very relationship to confirm the formula against measurements.

3. Do not let the tool and the server fight over the same CPU. If both compete for cores, you do not know whether you are measuring the server's limit or the tool's. In the lab the app is limited to 1 CPU and the tool gets 2, and we always verify with docker stats:

$ docker stats --no-stream --format '{{.Name}}  CPU={{.CPUPerc}}  MEM={{.MemUsage}}'

cpu-probe             CPU=101.20%  MEM=50.55MiB / 7.653GiB   # bo do: 101% tren HAN MUC 200%
sysdesign-lab-app1-1  CPU=82.72%   MEM=81.96MiB / 256MiB     # app:   83% tren han muc 100%

Read those two lines: the app is at 83% of the 1 CPU it was given — close to saturation. The tool is using 101% out of a 200% allowance, so roughly half its capacity. That lets us conclude that the throughput ceiling we measured is the server's ceiling, not the tool's. If the tool had hit 100% of its own allowance, every number would have to be thrown away.

4. One run is not a measurement. The three-repeat table in section 2.2 shows the same experiment, unchanged, producing collapse factors of 229× / 250× / 216×. Reporting "collapsed 250×" from a single run is the number of one run, not of the system. The honest form is a range: 200–250×.

2.5 Finding the knee point: where does your server saturate?

Now to answer Lesson 1's question. We raise the connection count step by step and record throughput alongside latency. The command for each data point:

for c in 1 2 4 8 16 32 64 128; do
  docker compose run --rm loadgen loadgen.js \
    --url http://app1:3000/fast -c $c -d 6 -w 2 --json
done

Real results on the machine listed at the top of the lesson:

Reading this table is the whole value of the lesson. Going from 16 to 128 connections, throughput drops 5% while p99 rises 8.4×. Past the knee, every additional concurrent request is pure cost — it only makes the queue longer. This is Lesson 1's non-linear wall, this time in measurements from a real machine.

Confirming Little's Law experimentally

The most convincing measurement comes from a different endpoint. Using /slow-sync?ms=5 — which occupies exactly 5 ms of CPU per request — the maximum service rate on one CPU must be $\mu = 1000/5 = 200$ req/s. Measured:

The last column is computed from $W = L/\lambda$ with $\lambda = 200$. The discrepancy against the measurements is under 0.3% at every level. Little's Law is not decorative theory — it is an equality that holds figure by figure on a real machine.

And note the most surprising thing in that table: throughput is flat at 200 req/s from a single connection onwards. With a CPU-bound handler on a single-threaded server, the server is already saturated at exactly one concurrent request. Every additional connection buys nothing at all — it only lengthens the queue and raises latency linearly.

Comparing against the simulation

The Traffic Lab below simulates exactly the situation we just measured. Pick the "1. A single server" topology and drag the load up: you will see the same shape as the table — throughput hits its ceiling and stops, while the queue and p99 keep climbing.

The limit is known — now what?

We have $\mu$ as a measurement, we know where the knee is, and we know a handler that blocks the event loop costs 200–250×. But however perfectly the code is written, one server still has a hard ceiling: one CPU, one process, one point of failure.

Lesson 3 puts nginx in front and multiplies the app into 3 replicas, then measures these very same numbers again. Along the way we meet two counter-intuitive results: a load-balancing algorithm whose "requests per node" chart looks perfectly balanced while its p99 is nearly twice as bad as the alternative, and the absence of graceful shutdown turning every deploy into a countable burst of 502s for real users.