Lesson 6: CDN & Edge Caching

Lesson 5 put the cache next to the app. This lesson puts it next to the user — and the reason isn't an architecture decision, it's a physical limit no amount of code can get around.

This lesson's lab builds a real edge layer with nginx and measures 25x throughput, with the origin only having to handle 4 requests out of 208,173. But the more valuable finding is the second measurement: a very common cache-key misconfiguration makes origin load jump 251x while every client-side metric stays unchanged — same throughput, same p50, same p99. It's completely invisible from the outside.

6.1 The limit no code can beat

In optical fiber, a signal travels at roughly $200,000$ km/s — slower than the speed of light in a vacuum because of the glass's refractive index. Converted into a more useful unit: 200 km per millisecond. A request has to go there and back, so the minimum round-trip time for a distance of $d$ (km) is:

$$RTT_{min} = \frac{2d}{200} = \frac{d}{100} \text{ ms}$$

That's 1 ms for every 100 km round trip, and that's the number for a straight line of fiber, no intermediate equipment, no queueing. Reality is always 2–3x worse, because cables don't run in straight lines, because every router on the path adds a bit, and because the link carries load.

Now combine this with what Lesson 4 section 4.2 measured: a new HTTPS connection needs 1 RTT for the TCP handshake, plus 1 more RTT for the TLS 1.3 handshake. With the origin in Virginia and a 250 ms RTT, a user in Hanoi spends roughly 500 ms before the first request is even sent. Your backend hasn't received anything yet.

6.2 CDN architecture, and how much the edge layer actually measures

A CDN is a network of many PoPs (points of presence) spread out geographically, each PoP a cache layer sitting in front of your origin. Users always talk to the nearest PoP, and "nearest" is decided by anycast: the same IP address is advertised from many locations, and BGP (Border Gateway Protocol — the protocol large networks on the internet use to exchange which routes reach where) routing automatically sends packets to the nearest one on the network, with no extra configuration from you.

The lab builds exactly the first tier: a second nginx, but very different from lb.conf and gw.conf — it stores responses instead of just forwarding them. Compared at 32 connections, 10 seconds, origin processing latency 30 ms:

Throughput is 25x higher, p50 drops 27x. But the last column is the one worth staring at: the origin processed 4 requests out of 208,173 — 0.002%. Everything else was returned straight from nginx's own cache, and the app has no idea any of this is happening.

6.3 Getting the caching headers right

The edge layer can't guess on its own how long content should be cached — it follows the headers the origin sends. This is the entire API surface between your app and the CDN, and it's smaller than most people think.

Revalidation: ETag and 304

When the TTL expires, the cache layer doesn't necessarily have to re-download everything. ETag is an identifier string for one specific version of the content — in the lab it's a hash of the body itself, so unchanged content means an unchanged ETag. When the TTL expires, the cache layer resends If-None-Match with the ETag it already has; if that ETag still matches (meaning the content hasn't changed), the origin returns 304 Not Modified with no body. Measured for real in the lab:

E=http://localhost:8082

etag=$(curl -s -D - -o /dev/null "$E/cacheable?key=b&ttl=60" \
  | grep -i '^etag:' | cut -d' ' -f2)
echo "ETag = $etag"                    # ETag = "7e9134f4"

curl -s -o /dev/null -w '%{http_code} %{size_download} byte\n' \
  "$E/cacheable?key=b&ttl=60"
# 200 62 byte                          ← downloads the full body again

curl -s -o /dev/null -H "If-None-Match: $etag" \
  -w '%{http_code} %{size_download} byte\n' "$E/cacheable?key=b&ttl=60"
# 304 0 byte                           ← only confirms "still valid", downloads nothing

For a 62-byte payload this saving is meaningless. For an 800 KB JavaScript file, every revalidation saves exactly 800 KB. That's why proxy_cache_revalidate on is in the lab's edge config: it turns "expired" from "have to re-download" into "just have to ask one question".

Serving stale content when the origin is down — and its limit

The STALE branch deserves its own measurement, because it's what keeps the site alive. Experiment: warm the cache with a 2-second TTL, then stop all three apps, wait for the TTL to expire, and call again:

# 1) warm the cache with TTL 2s
curl -sI "$E/cacheable?key=stale&ttl=2"     # X-Cache=MISS  36.5 ms
curl -sI "$E/cacheable?key=stale&ttl=2"     # X-Cache=HIT    3.6 ms

# 2) stop the whole origin
docker compose stop app1 app2 app3
sleep 3                                      # let the TTL expire

# 3) TTL expired + origin down -> what does edge do?
curl -sI "$E/cacheable?key=stale&ttl=2"     # 200  X-Cache=STALE     2.4 ms
curl -sI "$E/cacheable?key=stale&ttl=2"     # 200  X-Cache=UPDATING  2.5 ms
curl -sI "$E/cacheable?key=stale&ttl=2"     # 200  X-Cache=UPDATING  2.5 ms

# 4) a URL that was NEVER cached, origin still down
curl -sI "$E/cacheable?key=never-cached"    # 504  X-Cache=MISS   6.03 SECONDS

The three lines at step 3 are the best possible outcome: the origin is completely dead and the user still gets a 200 OK in 2.4 ms. UPDATING means nginx is trying to refresh in the background while still serving the old copy to the user — exactly the point of proxy_cache_background_update.

But step 4 is the line to remember. A URL that was never in the cache gives the edge nothing to return: 504 after 6.03 seconds. And that 6-second figure isn't random — it's 3 upstreams × proxy_connect_timeout 2s, the exact mechanism seen back in Lesson 3. A user waits 6 seconds and then gets an error, much worse than getting the error right away.

6.4 Cache key and Vary — where hit ratio quietly goes to 0

The cache key decides whether two responses count as "the same thing" or not. It's made up of the URL (nginx's default is $scheme$proxy_host$request_uri, meaning the entire query string is included) plus whatever headers the origin declares in Vary.

These two mechanisms are the two most common ways to destroy hit ratio, and neither one produces any error — the config still looks like "caching is on".

Measuring the damage: 251x more origin load, and completely invisible

Verified with numbers at load scale. Same content, but the load generator attaches a utm_source parameter drawn randomly from 1,000 values — exactly like real traffic from ad campaigns. Comparing two cache-key configs, 32 connections, 10 seconds:

Origin load differs by 251x. And the three prior columns — throughput, p50, p99 — are indistinguishable from each other. From the client's side, these two configurations are identical. No latency dashboard, no alert, no error tells you anything.

The number 1,003 is also worth reading closely: 1,000 distinct utm_source values produce exactly 1,000 distinct cache keys, one MISS per key. The cache is still "working" — it's just storing 1,000 copies of the same content, and none of them ever get reused.

The config that produces both behaviors above, one line apart:

location /cacheable {
    # RAW KEY: nginx's default, includes the ENTIRE query string.
    # ?utm_source=facebook => a separate cache entry for the exact same content.
    proxy_cache_key "$scheme$proxy_host$request_uri";
    proxy_pass http://edge_origin;
    include /etc/nginx/snippets/edge-common.conf;
}

location /norm/cacheable {
    # NORMALIZED KEY: only lists the parameters that ACTUALLY decide the content.
    # nginx has no loop, so this "allowlist" is spelling out every $arg_* we accept
    # by hand. Tracking params (utm_*, fbclid, gclid) are ignored simply because
    # they are NOT part of the key.
    proxy_cache_key "norm|$uri|key=$arg_key|ttl=$arg_ttl|vary=$arg_vary";
    proxy_pass http://edge_origin/cacheable$is_args$args;
    include /etc/nginx/snippets/edge-common.conf;
}

6.5 Purge, versioned URLs, and how to blank the page for every user

When content changes, there are two ways for users to see the new version: delete the old copy from the cache (purge), or change the URL so the new version lives at a different address. The second option is almost always better, for a very practical reason.

From that comes a very tidy rule for setting headers, broken down by resource type:

Reproducing the measurements yourself

cd blog/sysdesign/sysdesign-lab
docker compose --profile edge up -d
E=http://localhost:8082
purge() { docker compose exec -T edge sh -c 'rm -rf /var/cache/nginx/edge/*'; }

# --- MISS then HIT ---
purge
for i in 1 2 3; do
  curl -s -o /dev/null -D /tmp/h -w "%{time_total}s " "$E/cacheable?key=a&ttl=60"
  grep -i '^x-cache:' /tmp/h
done

# --- Vary: Cookie fragments the cache ---
purge
for c in user1 user2 user3 user1; do
  curl -s -o /dev/null -D /tmp/h -H "Cookie: sid=$c" \
    "$E/cacheable?key=e&ttl=60&vary=cookie"
  echo "sid=$c -> $(grep -i '^x-cache:' /tmp/h)"
done

# --- Origin load: RAW key vs NORMALIZED key (1000 random utm_source values) ---
purge && ./tools/cache-stats.sh reset
docker compose run --rm loadgen loadgen.js \
  --url "http://edge:8082/cacheable?key=a&ttl=60" \
  -c 32 -d 10 -w 3 --key-space 1000 --key-param utm_source --json
./tools/cache-stats.sh          # see how many requests the origin actually processed

purge && ./tools/cache-stats.sh reset
docker compose run --rm loadgen loadgen.js \
  --url "http://edge:8082/norm/cacheable?key=a&ttl=60" \
  -c 32 -d 10 -w 3 --key-space 1000 --key-param utm_source --json
./tools/cache-stats.sh          # compare: 1,003 vs 4

Summary

Geographic distance is a physical constant: 1 ms for every 100 km round trip, and a new HTTPS connection needs 2 RTTs before the request is even sent. There's no way to fix this with code — only to place the data closer. The lab's edge layer delivers 25x throughput and keeps 99.998% of requests away from the origin, using nothing but nginx and proxy_cache.

The three easiest mistakes all live in the header and cache-key area, and none of them produce an error: no-cache still caches (you need no-store to actually not store it); an ETag generated from a moment in time instead of from content means 304 never happens; and a cache key containing a tracking parameter makes origin load jump 251x while not a single client-side metric changes at all.

Finally, proxy_cache_use_stale keeps the site alive when the origin goes down — but only for URLs that were already in the cache. That turns cache coverage into part of your incident plan, and turns "flush the cache and then deploy" into something worth rethinking.

Lesson 7 leaves the cache tier and goes down to the data tier: replication. The lab will have a real PostgreSQL primary and replica, measure replication lag with pg_stat_replication, and reproduce by hand the bug every system with a replica runs into — a user changes their profile photo, reloads, and sees the old one.