Lesson 3: Scaling Out & Load Balancing

Lesson 2 measured one server's hard ceiling: the knee sits around 4 concurrent connections, and with a CPU-bound handler throughput stands still at 200 req/s from a single connection onward. However perfect the code, one process is still one process — and one point of failure.

This lesson puts nginx in front, multiplies the app into three replicas, then measures again. Two results turned out counter-intuitive enough that I re-ran them several times to be sure: a distribution algorithm whose chart looks completely balanced yet whose p95 is 340x worse, and graceful shutdown — which every piece of documentation recommends — turns out to make no measurable difference in the single most common case.

3.1 Scaling up and scaling out

Scaling up is giving the machine more power: more CPU, more RAM. Scaling out is adding more machines. The two directions are not equivalent, and picking the wrong one is one of the most expensive mistakes you can make.

Scaling up wins on simplicity: no distributed thinking required, no consistency lost, no extra network hop. But it has three hard limits:

• A physical ceiling. There is no such thing as an infinitely powerful machine.
• Non-linear pricing. A machine with double the performance usually costs more than double, especially at the high end.
• Still a single point of failure. This is the most serious limit and cannot be bought away — a dead 128-core machine still means a dead system.

Scaling out removes all three, but it comes with one non-negotiable prerequisite: the application must be stateless. Concretely: the process must not hold anything in RAM that a later request needs to read back — session data, a local cache, a counter, a half-finished computation.

3.2 L4 and L7: what a load balancer can "see"

The core difference between the two layers is not speed — it is how much information the load balancer can read, and the price paid to read it.

3.3 Distribution algorithms — and the measurement that made me re-run it

The four most common algorithms:

When every request costs the same and every replica is equally healthy, all four give near-identical results. The difference only shows up when there is skew — and in practice there is always skew: one replica just had a GC pause, one is running on a noisier host, one just restarted so its cache is still cold.

Experiment: one degraded replica

This is a deterministic and very real scenario: three replicas, one of which has its event loop occupied (I pump load into /slow-sync?ms=30 directly on app3, bypassing the load balancer). Then I measure /fast through the load balancer. Repeated 3 times per algorithm:

Slow down reading this table — it holds three things worth noticing:

• Throughput differs by roughly 40x (~170 req/s vs ~7,100 req/s). Same hardware, same source code, exactly one config line different.
• p95 differs by roughly 340x (~120 ms vs 0.35 ms). With round robin, about one in three requests lands on the blocked replica and has to wait its turn.
• But p50 is nearly identical (0.4–0.5 ms vs 0.23 ms). This is the dangerous part: if your dashboard shows median latency, this disaster is completely invisible.

Implementing the four algorithms

Just a few lines per algorithm. This is the real logic running in the series' simulator (sysdesign-sim-engine.js), so you can cross-check the simulation's behaviour against the lab:

// `alive` = list of replicas still alive. Each replica has: busy (in-flight count), queueDepth.
function pickReplica(alive, policy, rng) {
  if (alive.length === 0) return null;      // whole tier is dead => must drop, cannot hang
  if (alive.length === 1) return alive[0];

  switch (policy) {
    case 'round-robin': {
      const r = alive[this.rrCursor % alive.length];
      this.rrCursor++;
      return r;                              // does NOT read replica state at all
    }

    case 'least-connections': {
      let best = alive[0];
      for (const r of alive) {
        if (r.busy + r.queueDepth < best.busy + best.queueDepth) best = r;
      }
      return best;                           // reads state => avoids the congested node
    }

    case 'random':
      return alive[Math.floor(rng.next() * alive.length)];

    case 'random-two-choices': {
      // "Power of two choices": sample 2 replicas at random, keep the less busy one.
      // Cheap like random, but balance quality is close to least-connections.
      const a = alive[Math.floor(rng.next() * alive.length)];
      const b = alive[Math.floor(rng.next() * alive.length)];
      return a.busy + a.queueDepth <= b.busy + b.queueDepth ? a : b;
    }
  }
}

The four switch branches above are the entire difference between the algorithms: the last two (least-connections, random-two-choices) read replica state before picking, the first two (round-robin, random) have no idea which replica is busy at all. In nginx it is just one line inside the upstream block (the block declaring the group of backend servers nginx distributes load to):

upstream app_pool {
    # nginx defaults to round robin (nothing to declare for it).
    # Uncomment the line below to switch to least connections, then RE-MEASURE:
    least_conn;

    server app1:3000 max_fails=2 fail_timeout=5s;
    server app2:3000 max_fails=2 fail_timeout=5s;
    server app3:3000 max_fails=2 fail_timeout=5s;

    # Persistent connections to upstream: without this, nginx opens a new TCP
    # connection for EVERY request, and most of the latency you measure will
    # just be the internal TCP handshake cost.
    keepalive 64;
}

3.4 Health checks, failover and graceful shutdown

A load balancer is only useful if it knows which replicas are alive. There are two ways to know, and open-source nginx only has one:

• Active: the load balancer periodically calls /health. Early detection, pulling a node out before any request is affected. This is an nginx Plus feature — not present in the open-source build.
• Passive: counts real failures. max_fails=2 fail_timeout=5s means after 2 failures within 5 seconds, that node is temporarily removed. The important consequence: the first few requests after a node dies still have to fail — that is the price of detection.

But knowing which replica died is only half the story. The other half is graceful shutdown: when that replica stops on purpose (a deploy, a scale-down, an operator command), it should not exit immediately. The right sequence: mark itself unhealthy so the load balancer stops sending it new requests, then drain — wait for in-flight requests to finish — and only then exit the process. The Unix signal that usually triggers this is SIGTERM (a request to stop gracefully, giving the process a chance to clean up before exiting), as opposed to SIGKILL (an immediate kill, no chance to clean up). docker compose stop sends SIGTERM first, and only escalates to SIGKILL if the process has not exited after a grace period.

Experiment: stopping a replica under load

Running load against /fast through the load balancer with 8 connections, and at second 5, docker compose stop app3. I expected to see a burst of 502 errors. The actual result was completely different:

Two findings, and both went against what I predicted before measuring.

First: zero errors. The cause sits right in the proxy_next_upstream config — nginx automatically retries the request on another upstream. nginx's log shows each case clearly:

172.21.0.6 "GET /fast HTTP/1.1" status=200 \
  to=172.21.0.2:3000, 172.21.0.4:3000 \
  ut=2.001, 0.000  rt=2.001  tries=504, 200

# How to read this line:
#   to=   two upstreams => already retried
#   ut=   first upstream took 2.001s (proxy_connect_timeout expired), second took ~0s
#   tries=504, 200  => first attempt timed out, second attempt succeeded
#   status=200      => CLIENT NEVER SAW AN ERROR, only extra latency

This is good news and bad news at once. Good news: the retry saves availability. Bad news: it turns lost availability into lost latency, and the size of that loss is exactly the proxy_connect_timeout you configured — 2 seconds here. If you only watch the error rate, you would conclude "the deploy went completely smoothly" while ~30 users just waited 2 seconds.

Second: graceful shutdown makes no measurable difference. 27 versus 31 retried requests is within noise. This sounds like it contradicts every piece of operations documentation, but the reason is simple: /fast finishes in 0.2 ms, so at the moment SIGTERM arrives there is almost never an in-flight request left to drain. The app's log confirms it: "marking unhealthy, waiting for 0 in-flight requests..."

So when does graceful shutdown actually help? Re-measuring with a long request

Repeating the exact same experiment but switching to /slow-async?ms=500 — a request long enough that there are always a few in flight when the node is stopped:

Now the difference is clear: 861 ms vs 2,515 ms — the tail latency is roughly 2.9x worse. Five in-flight requests get cut off mid-flight; nginx has to retry them on a different upstream, and most of that cost is the 2-second wait for proxy_connect_timeout to expire before retrying, plus the time to redo the unfinished work.

3.5 Sticky sessions and the price of local state

When the app keeps sessions in RAM, the fastest fix is forcing every user back to the exact same replica every time — a sticky session. In nginx that is just one line: ip_hash;. It works, and that is exactly the problem.

The correct fix is moving sessions out to a shared store — Redis, or a signed token so nothing needs to be stored at all. At that point the app is genuinely stateless and every problem above disappears. Lesson 5 builds exactly that layer.

Reproduce the measurements yourself

# 1) Bring up nginx + 3 replicas
cd blog/sysdesign/sysdesign-lab
docker compose --profile lb up -d
sleep 10                      # IMPORTANT: let all 3 replicas finish warming up

# 2) Check the distribution (round robin => counts should be roughly equal)
for i in $(seq 1 30); do curl -s http://localhost:8080/whoami; echo; done \
  | grep -o '"instance":"app[0-9]"' | sort | uniq -c

# 3) DEGRADE exactly one replica: hit app3 directly, bypassing the LB
docker compose run --rm -d --name hog loadgen \
  loadgen.js --url "http://app3:3000/slow-sync?ms=30" -c 3 -d 25 -w 0

# 4) Measure /fast through the LB while app3 is blocked
docker compose run --rm loadgen \
  loadgen.js --url http://lb:8080/fast -c 6 -d 10 -w 3

# 5) Switch to least_conn, then RE-MEASURE. Write the file atomically, not with sed -i:
sed 's/^    # least_conn;/    least_conn;/' nginx/lb.conf > /tmp/lb.new \
  && mv /tmp/lb.new nginx/lb.conf
docker compose restart lb
# CONFIRM the config actually took effect before trusting the next measurement:
docker compose exec lb grep -c '^ *least_conn;' /etc/nginx/conf.d/default.conf

# 6) Count requests nginx retried, and requests that took >= 1 second
docker compose logs --no-log-prefix lb | grep -c 'to=[^ ]*,'
docker compose logs --no-log-prefix lb | grep -oE 'rt=[0-9.]+' \
  | awk -F= '$2>=1{n++} END{print n+0}'

Three questions to answer for yourself after running this: is the count distribution balanced (it should be), how far apart is p95 between the two algorithms, and does p50 reflect the problem at all (it should not).

Comparing against the simulation

The Traffic Lab below lets you switch distribution algorithms and kill a replica right on the page. Pick the "2. LB + multiple app servers" topology, set the load high enough, then click "Kill 1 app server" to watch the queue pile onto the remaining replicas and utilization spike.

Three replicas now — but what are they fighting over?

We removed the single point of failure and multiplied throughput. But notice one thing running through this entire lesson: all three replicas read from the exact same data source. Scaling the stateless tier is the easy part — it only pushes the bottleneck one layer down.

Lesson 4 extends nginx from a load balancer into an API gateway — routing by path, TLS termination, and one specific security pitfall: why you must never trust the X-Forwarded-For header a client sends you.