Lesson 17: Failure Modes & Resilience

Lesson 16 just finished building the measurement toolkit. This lesson uses it to look at something hard to believe if you only hear it described: a system taking itself down with traffic it generates itself, long after the root cause is gone.

Four measurements from the lab. First, a three-tier chain where each tier retries three times turns 30 user requests into 810 hits on the leaf service — exactly 27 times, and no tier did anything wrong. Second, backoff without jitter bursts traffic to 19.6 times the average; adding jitter brings that down to 5.3 times. Third, a circuit breaker cuts calls into a sick dependency from 17 down to 7 and turns 5,291 ms of waiting into 2,173 ms. Fourth, a timeout placed in the wrong spot makes the server do the full 50,000 ms of wasted work — 100 complete responses that nobody receives.

17.1 Cascading failure: when a system takes itself down

The propagation mechanism is always the same and doesn't depend on the technology. A dependency slows down — not dead, just slow. Whoever calls it holds a connection and a thread open, waiting. Connections held longer means the pool drains faster. A drained pool means the caller itself starts slowing down, and whoever calls it starts holding connections open longer too. Within a few dozen seconds, a small incident at the bottom tier has propagated all the way up to the user.

The most dangerous trait of a cascade is that it is self-sustaining: after the root cause has been fixed, the system keeps falling over, because what's keeping it down now is the retry traffic piling up. And the biggest amplifier of that traffic is something that looks harmless in code review.

// This code shows up at EVERY layer, and at every layer it looks reasonable:
// "if the call fails, retry a few times, the network might just be flaky".
for (let attempt = 1; attempt <= 3; attempt++) {
  const r = await callDownstream();
  if (r.ok) return r;
}
throw new Error('downstream failed');

Nobody catches the problem reviewing that snippet, because the problem isn't in the snippet — it lives in the product of the whole chain. The lab builds three real tiers, each running exactly that code, then counts how many times the leaf service gets called:

$ ./tools/resilience-test.sh amplify 30 0
  30 request nguoi dung · chuoi 3 tang · moi tang thu lai 3 lan · budget=0
     app1 leafHits=810 retry=60  budgetDenied=0
     app2 leafHits=0   retry=540 budgetDenied=0
     app3 leafHits=0   retry=180 budgetDenied=0
  (30 x 27 = 810 lan dap vao service tan cung)

810 calls for 30 requests. The factor of 27 = 3 × 3 × 3, and it lands right when the leaf service is weakest — because that very weakness is what's generating the retries in the first place. This is a pure positive feedback loop: the weaker it gets, the more it gets called; the more it gets called, the weaker it gets.

17.2 Getting retries right: conditions, backoff, and jitter

Three conditions must hold at the same time for a retry to be safe. Missing any one breaks it:

The third condition is the one most often done halfway: plenty of systems have backoff but no jitter. The consequence is very concrete — when a service dies and comes back, every client that hit the error at the same moment also waits exactly 100 ms, then exactly 200 ms, then 400 ms. Every one of those points is a synchronized wave.

The lab fails 200 clients at once and lets them retry 4 times, both modes using the same backoff formula:

The column worth staring at is the burst factor — how many times the peak is above the average. No jitter gives 19.6 times: a server that just recovered has to absorb a spike nearly 20 times the average load, and that spike is usually enough to kill it again, producing a crash–recover–crash loop. Adding jitter brings the factor down to 5.3 and the peak down to a third.

17.3 Timeouts and deadlines: don't do work for a request that's already dead

Every network call needs a timeout. That sentence sounds obvious enough to skip, but the default in a great many HTTP libraries is infinite — meaning your default is also infinite, unless you've explicitly set one. A call with no timeout holds a connection open indefinitely, and that is exactly the first step of the cascade mechanism in section 17.1.

But having a timeout isn't enough on its own; timeouts must shrink with depth. If the outer tier waits 1 second while the inner tier waits 3, the outer tier gives up first while the rest of the chain keeps grinding away on a request nobody is waiting for anymore. The lab measures exactly that situation: the client gives up after 100 ms, the server works for 500 ms.

{
  "role": "deadline",
  "soRequest": 100,
  "timeoutPhiaClient": 100,       // client gives up, all 100 requests
  "serverVanLamViec": 100,        // server STILL does all 100 x 500ms of work
  "phanHoiKhongAiNhan": 100,      // 100 complete responses, nobody receives them
  "congToiMs": 50000              // 50 seconds of CPU and I/O for nobody
}

The scariest part of this number is that it's invisible on the dashboard. From the server's point of view, those 100 requests completed normally, no errors, processing time exactly as designed. From the user's point of view, all 100 failed. Both sides are looking at the same system and seeing opposite truths — and the side with the dashboard is the one looking at it wrong.

The fix is called deadline propagation: instead of every tier setting its own timeout, the first tier computes an absolute point in time by which the request must finish, and passes that point downward. Every tier checks it before doing work, and bails immediately if it's already past. The result: nobody does work for a request that's already dead.

17.4 Circuit breakers and bulkheads

Retrying assumes an error is temporary. Once a dependency is genuinely broken, that assumption is wrong — and every retry just adds another wait before receiving the same error again. A circuit breaker recognizes that and stops calling entirely for a while, giving the dependency room to breathe and saving itself from sitting there waiting.

The lab kills a dependency for the first 5 seconds (every call hangs 300 ms then returns 503), then lets it recover:

Read the table both ways. For the dependency: calls hitting it drop 59% exactly when it needs quiet the most. For the caller itself: 5.3 seconds of waiting turns into 2.2 seconds, and 407 requests get an answer immediately instead of hanging — meaning threads and connections get freed up, which is exactly where the cascade gets stopped.

Bulkheads solve a sibling problem: even with a breaker, if every dependency shares one connection pool, a slow dependency can still drain the pool before the breaker has a chance to open. The fix is to give each dependency its own pool — exactly like the watertight compartments on a ship the name evokes: one flooded compartment doesn't sink the whole vessel.

17.5 Graceful degradation and chaos engineering

The three sections above are all about not making the system worse. This one is about making it still usable once a part is already broken. The principle: a degraded result is almost always better than an error page.

• A product page loses its personalized recommendations — you can still buy something.
• Cached data that's 10 minutes stale instead of an error — on most screens, nobody notices.
• Hide the view count when the stats service is dead — nobody's there for that number.
• A safe default value instead of calling the config service — as long as that default is tested.

The precondition is identical to prioritized load shedding in Lesson 13: you need to know in advance which paths have to survive no matter what, and which parts are fine to switch off. That's a product decision, not a technical one, and it has to be made before the incident.

Reproduce the measurements yourself

cd blog/sysdesign/sysdesign-lab

# --- Retry amplification: 3-tier chain, each tier retries 3 times ---
./tools/resilience-test.sh amplify 30 0   # 810 hits on the leaf service (27x)
./tools/resilience-test.sh amplify 30 1   #  69 hits with a 10% retry budget enabled (11.7x fewer)

# --- Retry storm: with and without jitter ---
./tools/resilience-test.sh jitter
#   no jitter: peak 185/20ms · mean 9.4  · burst factor 19.6x · lasts 1,694ms
#   jitter   : peak  65/20ms · mean 12.3 · burst factor  5.3x · lasts 1,302ms

# --- Circuit breaker: dependency dies for 5s then recovers ---
./tools/resilience-test.sh breaker
#   no breaker: 17 calls downstream · 5,291ms wasted waiting ·   0 immediate rejections
#   breaker   :  7 calls downstream · 2,173ms wasted waiting · 407 immediate rejections

# --- Timeout placed in the wrong spot ---
./tools/resilience-test.sh deadline
#   100 client-side timeouts · server STILL does all 100 · 50,000ms wasted work

In summary

Cascading failure isn't a type of error, it's an operating mode a system falls into, and it's self-sustaining after the root cause is gone. The fuel that feeds it is retries: three tiers, three retries each, turning 30 requests into 810 hits on the leaf service — exactly when that service is weakest. No tier did anything wrong; the factor of 27 only exists at the whole-system level, so the only way to catch it is counting how many times the leaf service gets called per request.

Three fixes, each solving a different part. Retry budgets stop the amplification itself: 810 down to 69, 11.7 times fewer, because it limits by ratio rather than by count. Jitter stops the synchronized wave: burst factor 19.6 times down to 5.3, and it finishes sooner too. Circuit breakers stop the waiting: 5.3 seconds of waiting becomes 2.2 seconds, 407 requests get an answer immediately instead of hanging — and that's exactly where threads and connections get freed.

On timeouts, the number worth remembering is 50,000 ms of wasted work: 100 complete responses nobody receives, and nothing on the server's dashboard shows it — from its side, every request completed normally. Timeouts must shrink from outer to inner, and the correct way is propagating an absolute deadline rather than letting each tier pick its own number.

Finally, a lesson that came from writing this lab itself: the first version of the circuit breaker was missing the "probe fails → reopen immediately" branch, and it got stuck in half-open forever — rejecting every request permanently even after the dependency was healthy again, without throwing a single error. Resilience mechanisms are code too, and code has bugs; the difference is this kind of bug only shows up exactly when you need it most. That is the entire argument for chaos engineering.

Lesson 18 is the last one: put it all together into a real system, run it, measure it, find the bottleneck, fix it, then measure again — with before-and-after numbers for every step, so it's clear which steps were worth it and which just added complexity.