Lesson 1: Latency, Throughput & Queueing Theory

Every discussion about system design eventually reduces to two numbers: latency and throughput. That sounds simple, but it is where the misunderstanding starts — because most of us were trained to think linearly, while any system with a queue behaves non-linearly, and unforgivingly so.

This lesson establishes four foundations reused across the following 17: telling latency apart from throughput, memorising the orders of magnitude of different delays, understanding why the mean is a deceitful number, and finally the formula that explains why a server "with 30% of its CPU idle" can nonetheless be dying.

1.1 Latency and throughput: two quantities, not one

Latency is the time to serve one request, measured in units of time. Throughput is the number of requests served per unit of time, measured in req/s. They are related but neither follows from the other, and more importantly: optimising one usually harms the other.

The clearest example is batching. Collecting 100 writes and pushing them to the database at once raises throughput sharply, because the fixed cost per write is shared out. But the first request to arrive has to wait for the other 99 before it is processed — its latency goes up. Compression is the same: it saves bandwidth (more throughput on the same link) but adds compress/decompress time to every request.

The last row holds a subtle point that is very often misread: adding servers does not make a single request finish sooner. If a handler needs 20 ms of CPU it still needs 20 ms whether you have 1 server or 100. Extra servers only reduce queueing time. On an idle system, adding servers barely improves latency at all.

1.2 Latency orders of magnitude: the table that shapes every architecture

This is probably the most important table a system designer needs by heart. Not to recall each figure exactly, but to recall the distances between them — because those distances are what decide the architecture.

Read this table with application in mind. RAM is roughly 1.5 million times faster than an intercontinental round trip. That single fact explains three classic architectural decisions:

• Why caching wins almost always: turning a network query (~1 ms) into a RAM read (~100 ns) saves a factor of 10,000. No algorithmic optimisation offers a ratio like that.
• Why CDNs exist: when the user is half a world away from the server, 150 ms is a physical limit — covered properly in Lesson 6. The only remedy is to move the data closer to them.
• Why the "N+1 query" is a sin: 100 sequential queries × 1 ms = 100 ms, while merging them into one query might take 3 ms. The problem is not the database but the number of network round trips.

1.3 Tail latency: why the mean is a deceitful number

The latency distribution of a real system is almost always right-skewed: most requests are fast, a small share are markedly slow, and that long tail drags the mean along. The result is that the mean represents nobody — neither the typical experience nor the worst one.

So we use percentiles instead. Saying "p99 = 420 ms" means 99% of requests are faster than 420 ms and 1% are slower. For a service taking 10 million requests a day, that "1%" is 100,000 occasions of a user having a bad experience — every day.

Why a service's p99 becomes the user's p90

This is what makes tail latency far more dangerous than it first appears. A modern web page usually has to call several services to render one screen. If each call is independent and the probability of one call being slow is $p$, then the probability that the user hits at least one slow call when the page makes $n$ calls is:

$$P(\text{slow}) = 1 - (1 - p)^n$$

Here $p$ is the probability of a single call exceeding the slow threshold (for example $p = 0.01$ at the p99 threshold), and $n$ is the number of services the page must call. Substituting:

This is exactly why large systems track out to p99.9, and why splitting into many services makes the tail latency problem markedly harder — a point we will put concrete numbers on in Lesson 15.

1.4 Little's Law and queueing theory

Now for the part that explains most of what feels "strange" when operating systems. It starts with an equality that is surprisingly simple — Little's Law:

$$L = \lambda W$$

Here $L$ is the average number of requests inside the system, $\lambda$ (lambda) is the arrival rate (req/s), and $W$ is the average time each request spends inside the system (seconds). This holds for every stable system, with no assumptions at all about distributions.

A direct application: if your service takes 500 req/s and each request averages 40 ms, then the number being processed concurrently is $L = 500 \times 0.04 = 20$. That is precisely the number you need to size a connection pool or a worker count — not a guess.

Utilisation and the non-linear wall

Let $\mu$ (mu) be the server's maximum service rate (req/s). Utilisation is:

$$\rho = \frac{\lambda}{\mu}$$

To go further we need a concrete queueing model. The simplest is M/M/1, and those three symbols read left to right as: requests arrive randomly, service time is random, and there is one server.

With that model, the average time a request spends in the system is:

$$W = \frac{1}{\mu - \lambda}$$

Look at the denominator. As $\lambda$ approaches $\mu$, the denominator approaches 0 and $W$ approaches infinity. This is not a malfunction — it is the normal behaviour of every queue.

Concrete figures for the time spent waiting in the queue (excluding service time):

Watch it happen: the single-node Traffic Lab

Reading a table is one thing; watching the curve go vertical under your own hand is another. The demo below runs a real discrete-event simulation (not a pre-rendered animation): drag the load slider and watch p99 alongside the queue length. The blue line is the theory, $W = 1/(\mu-\lambda)$, and the amber points are the simulated measurements — the two must agree, and that agreement is also how the series' engine is validated.

The source: computing percentiles and simulating M/M/1

The two functions below are the real code running in the series' engine. The percentile function uses the "nearest-rank" method — the same one common load-testing tools report — so your numbers are comparable with theirs.

// Tinh phan vi theo phuong phap nearest-rank (khong noi suy).
// Luu y: KHONG sua doi mang dau vao - da tung co bug vi ham sort() tai cho.
function percentile(values, p) {
  if (!values || values.length === 0) return 0;
  const sorted = [...values].sort((a, b) => a - b);
  if (p <= 0) return sorted[0];
  if (p >= 100) return sorted[sorted.length - 1];
  const rank = Math.ceil((p / 100) * sorted.length);
  return sorted[Math.min(sorted.length - 1, Math.max(0, rank - 1))];
}

// Ly thuyet hang doi M/M/1. Tra ve null khi rho >= 1: hang doi tang vo han,
// KHONG co trang thai on dinh - dung tra ve mot con so gia o day.
function mm1Theory(lambda, mu) {
  const rho = lambda / mu;
  if (rho >= 1) return { rho, W: null, Wq: null, L: null, stable: false };
  const W = 1 / (mu - lambda);      // tong thoi gian trong he (giay)
  const Wq = rho / (mu - lambda);   // rieng thoi gian CHO trong hang doi
  const L = lambda * W;             // dinh luat Little
  return { rho, W, Wq, L, stable: true };
}

// Doi chieu nhanh: mu = 50 req/s (moi request 20ms)
for (const lambda of [25, 35, 45, 49]) {
  const t = mm1Theory(lambda, 50);
  console.log(`rho=${t.rho.toFixed(2)}  W=${(t.W * 1000).toFixed(1)}ms  L=${t.L.toFixed(2)}`);
}
// rho=0.50  W=40.0ms  L=1.00
// rho=0.70  W=66.7ms  L=2.33
// rho=0.90  W=200.0ms L=9.00
// rho=0.98  W=1000.0ms L=49.00   <-- chi tang lambda tu 45 len 49 ma W gap 5 lan

Those four result lines are the whole lesson of this section in miniature: raising load from 45 to 49 req/s — only 9% — takes time-in-system from 200 ms to 1000 ms, a factor of 5.

1.5 Back-of-envelope estimation

The final skill of this lesson: turning a requirement stated in words into the numbers you need to start designing. This step comes before drawing any diagram, because without it every later choice is guesswork.

Worked through on a concrete problem: 10 million daily active users (DAU).

# 1) DAU -> RPS trung binh
10.000.000 DAU x 20 request/nguoi/ngay = 200.000.000 request/ngay
200.000.000 / 86.400 giay              = ~2.315 req/s   (trung binh)

# 2) He so dinh: luu luong KHONG deu trong ngay
#    Thuc te thuong dat 2-5 lan trung binh vao gio cao diem.
2.315 x 3                              = ~7.000 req/s   (dinh)

# 3) Ti le doc/ghi: dinh hinh toan bo kien truc
#    Gia dinh 100:1 (mang xa hoi, tin tuc, thuong mai dien tu deu quanh muc nay)
doc:  ~6.930 req/s   -> cache + read replica giai quyet duoc (Bai 5, Bai 7)
ghi:  ~70 req/s      -> mot primary con thua suc, CHUA can sharding (Bai 8)

# 4) Dung luong
#    Moi request ghi ~2 KB payload
70 req/s x 2 KB x 86.400              = ~12 GB/ngay
                                       = ~4,4 TB/nam  -> can nghi ve archival

# 5) Bang thong ra
#    Moi response doc ~10 KB
6.930 req/s x 10 KB                    = ~69 MB/s = ~550 Mbps  -> CDN (Bai 6)

Those five steps take under two minutes and have already eliminated a great many wrong options. Most notably: at 70 writes per second, sharding is unnecessary — a decision that saves months of effort and a permanent tax in complexity. Skip the estimation step and it is very easy to start with a distributed architecture for a problem one server solves.

One question still unanswered

This entire lesson revolves around $\mu$ — the maximum service rate. The tables, the curve and every estimate depend on it. But we have not said what $\mu$ actually is for your server.

That number cannot be read from documentation and cannot be guessed from the machine's specification. It has to be measured. Lesson 2 stands up a real Node app in Docker, runs increasing load, and finds the point where throughput stops rising while latency explodes — the wall you have just seen in theory, this time in numbers from your own machine. Along the way we will also see a handler written the wrong way collapse throughput by a factor of 195.