Lesson 16: Observability — Metrics, Logs & Tracing

Lesson 15 just split a use case into four services and created a new problem: once a request crosses four processes, no single process sees its whole journey. Which hop ate the time? Where did it break? Without an answer to those two questions, every resilience mechanism in Lesson 17 is just guessing.

This lesson's lab builds all three pillars by hand in app.js — histogram, correlation ID, span — no library. Three measurements stand out. First, with an incident that makes 1% of requests 300 ms slower, the measured p99 is 0.90 ms — it completely misses the incident; only p99.9 reveals the 474 ms. Second, adding exactly one user_id label makes the number of time series jump from 2 to 49,317 and memory from 330 bytes to 8.4 MB — in just 8 seconds of load, on one instance. Third, forgetting to propagate the correlation ID across one hop drops the waterfall from 4 spans to 1 span, and all the time gets blamed on the wrong service.

16.1 Monitoring answers old questions, observability answers new ones

These two words get used interchangeably, but they describe two very different capabilities. Monitoring is having answers ready for questions you already knew you would ask: how much CPU, what's the error rate, did we cross a threshold. It produces dashboards and alerts. Observability is the ability to answer a question you never anticipated — exactly the kind that shows up in every real incident, because if you had anticipated it, you would already have fixed it.

The three pillars below differ along three dimensions at once: granularity, cost, and the kind of question they can answer. Getting any one of those dimensions wrong leads to a big bill and a system you still cannot debug.

16.2 Getting metrics right: RED, USE, and two pitfalls

Two sets of metrics cover almost every need. RED per service: Rate, Errors, Duration (the distribution of latency). USE per resource: Utilization, Saturation (queue depth — Lesson 1 already showed this is the real early-warning signal), Errors. A service with full RED per route and full USE per resource already answers most of the questions you'll ask.

But the D in RED has to be a distribution, not a single number. The lab reproduces the most typical shape of incident — the vast majority of requests fast, a small fraction 300 ms slower — and asks: which percentile can actually see that incident?

The p99 column has a cliff edge between 1% and 2%: from 0.90 ms up to 300 ms. The reason is simple once stated: p99 is the threshold below which 99% of requests fall, so when exactly 1% of requests are slow, that threshold sits right at the edge and can land on the fast side. p99 is blind to an incident that affects exactly 1% of users. At 1%, only p99.9 sees it — the app's histogram measures p99.9 = 474 ms.

The mean column is bad in a different way: it climbs steadily (1.61 → 3.06 → 6.28 → 15.60) so it looks informative, but no request actually took 3.06 ms. At 1% slow, the mean is 23 times higher than the median. It doesn't describe anyone's experience: not the normal user's (0.13 ms), not the affected user's (300 ms) either.

The second pitfall of metrics is not about values but about labels. Every combination of label values is its own time series, and every time series costs its own memory. The lab runs the same load with two different label configurations, 50,000 distinct users, over 8 seconds:

16.3 The correlation ID: one header, and where it gets dropped

Structured logging means every line is an object with a schema instead of a free-form string, so you can query by field instead of writing a regex at 3 a.m. But the single most important field in it is not the error message — it's the correlation ID: an identifier generated at the first hop (usually the gateway, Lesson 4) that travels with the request through every service.

// Generated at the first hop, reused if already present. These 3 lines are the whole "generate" part.
function corrIdOf(req) {
  return req.headers['x-correlation-id'] || `c-${INSTANCE}-${Date.now().toString(36)}-${seq++}`;
}

// Returned to the client: when they report a bug, they attach this ID and you find it instantly.
res.setHeader('X-Correlation-Id', corrId);

// And here is the part that gets forgotten — PROPAGATE it when calling another service:
function callPeer(peer, path, corrId) {
  const headers = corrId ? { 'X-Correlation-Id': corrId } : {};
  return new Promise((resolve) => {
    http.get({ host, port, path, headers }, ...);
  });
}

Everyone gets the "generate" part right. The "propagate" part is where it breaks, and it breaks silently: no error, every request still runs, it's just that the trace gets cut into two pieces that can't be joined back together. The lab measures the consequence by running the same three-hop chain with propagate=1 and propagate=0:

$ ./tools/observe-test.sh trace

  --- propagate=1        (CO truyen correlation ID)
  so chang: 4
    app1/traced-chain        292.52 ms   52.0%
    app2/slow-async          256.98 ms   45.7%     (!) thu pham hien ra ngay
    app3/slow-async            7.46 ms    1.3%
    app1/slow-async            5.71 ms    1.0%

  --- propagate=0        (QUEN truyen)
  so chang: 1
    app1/traced-chain        289.32 ms  100.0%     (!) quy HET cho app1

Read the last line closely: when the ID isn't propagated, the system reports app1 accounting for 100% of the time. That is not "missing information" — it is wrong information, and it's wrong in the most dangerous direction, pointing straight at an innocent service. A team reading that waterfall will go optimize app1 while the real culprit (app2, eating 45.7% of the time) sits untouched.

16.4 Tracing: spans are scattered, someone has to collect them

A trace is a request's entire journey; each span is one leg of it, with a start time, a duration, and a parent span. Stitched together they form a waterfall — which answers "which hop ate the time" by showing you, not by making you guess.

There's one architectural detail the lab is forced to confront, and it's worth calling out: each service's spans live in the memory of that service alone. No process naturally sees the whole picture. That's why every real tracing system needs a collector — in this lab it's the /trace-all endpoint, which asks every instance and stitches the spans together. This is exactly the role a collector plays in real systems, and it's why tracing always drags in one more piece of infrastructure you have to operate.

This table connects straight back to section 16.2: if your incident only affects 1% of requests, then head-based sampling at a 1% rate will keep exactly 1% of that 1% — one in ten thousand. You'll have metrics reporting a bad p99.9 but no trace at all of a slow request to look at. That's why mature systems tend to use tail sampling to keep exactly the traces worth keeping: the slow ones and the erroring ones.

16.5 SLI, SLO, and error budget: turning reliability into a decidable number

These three concepts are often treated as management process, but they solve a genuinely technical problem: deciding when to stop shipping features and go fix reliability. Without them, that debate happens by gut feeling and is usually won by whoever talks loudest.

• SLI — a metric measured from the user's point of view, shaped like "good events over total events." Example: the fraction of requests that succeed and finish under 250 ms.
• SLO — a numeric target for that SLI, e.g. 99.9%.
• Error budget — the share you're allowed to fail: 100% − SLO. With a 99.9% SLO, the budget is 0.1%, about 43 minutes a month.

The lab computes the SLI directly from the histogram — the way real systems do it, and only possible because there's a histogram rather than an average:

The middle row is the one worth remembering, and it closes the loop on the whole lesson: the exact incident that p99 cannot see (1% slow, p99 = 0.90 ms) burns 8.7 times that month's error budget. If you only alert on p99, you'll sleep soundly while the whole quarter's budget evaporates.

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab

# --- Which percentile actually SEES the incident? ---
./tools/observe-test.sh percentile
#   0.5% slow -> p99   0.49ms      1% slow -> p99   0.90ms   (p99.9 = 474ms)
#     2% slow -> p99 300.01ms      5% slow -> p99 303.05ms
#   p99's cliff edge sits between 1% and 2%.

# --- What happens if we add one user_id label? ---
./tools/observe-test.sh cardinality
#   route,status        ->      2 time series ·     330 bytes
#   route,status,user   -> 49,317 time series · 8,855,160 bytes

# --- Correlation ID: propagated versus dropped ---
./tools/observe-test.sh trace
#   propagate=1 -> 4 spans, app2 eats 45.7% of the time
#   propagate=0 -> 1 span, 100% blamed on app1 (WRONG, not just missing)

# --- SLI and error budget computed from a real histogram ---
./tools/observe-test.sh slo
#   1% slow -> SLI 99.034% against a 99.9% target -> 865.5% of the month's budget burned

In summary

The three pillars don't replace each other: metrics tell you that something's wrong, traces tell you where, logs tell you why. Their cost grows in exactly that order, and each one has its own pitfall.

The pitfall of metrics is assuming p99 is enough. The measurements show the opposite: with an incident affecting exactly 1% of users, p99 = 0.90 ms — completely blind — while p99.9 = 474 ms and that month's error budget was burned 8.7 times over. Pick a percentile based on the fraction of users you need to protect, not out of habit. And the mean is bad in a different way: 23 times higher than the median while describing nobody's actual experience.

The second pitfall of metrics is labels, and it costs real money: adding exactly one user_id label makes the number of time series jump from 2 to 49,317 — over 24 thousand times — in just 8 seconds on one instance. Labels must have a finite, known-in-advance set of values.

On tracing, the most expensive lesson isn't technical — it's the consequence of one small omission: forgetting to propagate the correlation ID doesn't leave you missing data, it gives you wrong data — 100% of the time blamed on an innocent service, while the real culprit eats 45.7%. And the spot it's most often forgotten is the queue boundary, exactly the hardest place to debug.

Lesson 17 uses this exact measurement setup to look at something you'd never believe without measuring it: a system taking itself down with its own retry traffic, where three layers each retrying three times produce twenty-seven times the load on the weakest service.