Lesson 4: Reverse Proxy & API Gateway

Lesson 3 put nginx in front of three replicas and gave it exactly one job: splitting load. But the component sitting in that spot — where every request must pass through — can do far more: terminate TLS, route by path, serve static files, reject requests missing a token, rate-limit. This lesson keeps the same nginx but hands it those jobs, then measures what each one is actually worth.

Two measurements in this lesson went against what I predicted while writing the outline. First: the advice "let the gateway serve static files so the app doesn't have to" turned out to make the gateway 2.6x slower than forwarding the request to the app — until adding exactly one config directive. Second: "TLS costs about 10%" is a number I measured myself, and one that gets quoted a lot, but it only holds under one very specific condition; drop that condition and the cost becomes nearly 16x.

4.1 Three names for the same position

"Reverse proxy", "load balancer" and "API gateway" often get talked about as three different products. In reality they are three levels of responsibility assigned to the same position in the system: the hop standing between the client and the services. The same nginx process in this series' lab plays all three roles in turn.

One more easily-confused pair worth separating: forward proxy and reverse proxy. A forward proxy sits next to the client and represents the client (a company's outbound proxy to the Internet, say); a reverse proxy sits next to the server and represents the server. Same technology, opposite direction, and whoever configures it is different too.

This lesson's lab

The series' shared lab gets a new gw profile. The important design detail: the HTTP port (8081) and the HTTPS port (8443) include the exact same route file, so any measured difference between the two ports comes purely from TLS, with no configuration drift mixed in:

cd blog/sysdesign/sysdesign-lab

# 1. Generate a SELF-SIGNED cert for the gateway (lab only, never use it anywhere else)
mkdir -p nginx/certs
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout nginx/certs/lab.key -out nginx/certs/lab.crt \
  -subj "/CN=localhost"

# 2. Start the gateway + 3 app replicas
docker compose --profile gw up -d

# 3. Check that both ports are alive
curl -s  http://localhost:8081/gw-health      # -> gateway ok
curl -sk https://localhost:8443/gw-health     # -> gateway ok  (-k: skip cert verification)

# 4. Gateway strips the /api/ prefix before forwarding
curl -s http://localhost:8081/api/whoami
# -> {"instance":"app1", ...}   the app has NO idea the /api/ prefix even exists

If both curl commands in step 3 return gateway ok, the gateway is running correctly and ready for the measurements ahead. Step 4 confirms the single most important fact in this lesson: the gateway and the app speak two different paths — the client calls /api/whoami, the app only ever sees /whoami.

4.2 TLS termination and the truth about X-Forwarded-For

TLS termination means the gateway decrypts HTTPS and speaks plain HTTP to the services behind it. The reason is purely practical: the certificate lives in exactly one place (renew it once instead of N times), services don't need to know anything about TLS, and the gateway can read the content, which is what makes path-based routing possible in the first place.

How much does TLS cost?

Same route /api/fast, 8 concurrent connections, 8 seconds measured, three repeats — the only difference is whether the entry port is HTTP or HTTPS:

About 10% throughput, and the extra latency is far smaller than even a trivial database query. The conclusion looks obvious: TLS is cheap, turn it on. But this number is hiding a condition.

The 10% figure only holds with keep-alive

The lab's load generator defaults to persistent connections (keep-alive): open 8 connections and push hundreds of thousands of requests through them. What's expensive about TLS is the handshake — asymmetric key exchange, one or two network round trips — while the symmetric encryption that follows is very cheap. With keep-alive, the handshake cost gets divided across tens of thousands of requests and nearly disappears.

The load generator has a --no-keepalive flag that forces every request to open a new connection. Same route, same connection count:

Drop keep-alive, and HTTP loses 3x throughput because it has to redo the TCP handshake every time. But HTTPS loses nearly 16x compared to itself with keep-alive (20,400 req/s down to about 1,260 req/s), and p50 jumps from 0.36 ms to over 6 ms — 17x. That's the real shape of TLS cost: it doesn't live in every request, it lives in every connection.

What information does the app lose after decryption?

The TCP connection the app sees is a connection from the gateway, not from the client. So the app's socket.remoteAddress is always the gateway's IP, and the app no longer knows whether the client came in over HTTP or HTTPS. That information has to be forwarded through headers — and this is exactly where a very common vulnerability comes from.

The lab has a /client-ip endpoint that exists purely to demonstrate this. Run it for real, with a forged header attached:

curl -s -H 'X-Forwarded-For: 1.2.3.4' http://localhost:8081/api/client-ip

# {"instance":"app1",
#  "remoteAddress":"::ffff:172.21.0.5",              ← the GATEWAY's IP, not the client's
#  "xForwardedForRaw":"1.2.3.4, 192.168.65.1",       ← spoofed value + real IP
#  "trustProxyHops":0,
#  "trustedClientIp":"::ffff:172.21.0.5",
#  "naiveClientIp":"1.2.3.4"}                  ← WRONG: took spoofed value verbatim

# ---- Fix 1: gateway OVERWRITES the header (route /safe-api/) ----
curl -s -H 'X-Forwarded-For: 1.2.3.4' http://localhost:8081/safe-api/client-ip

# "xForwardedForRaw":"192.168.65.1"                  ← spoofed value wiped out
# "naiveClientIp":"192.168.65.1"                     ← correct no matter how you read it

# ---- Fix 2: app counts from the right ----
# Set TRUST_PROXY_HOPS='1' for all three apps in docker-compose.yml (default is '0'),
# then RECREATE the containers - `restart` does not reload environment variables:
#   docker compose --profile gw up -d --force-recreate app1 app2 app3
curl -s -H 'X-Forwarded-For: 1.2.3.4' http://localhost:8081/api/client-ip

# "xForwardedForRaw":"1.2.3.4, 192.168.65.1",
# "trustProxyHops":1,
# "trustedClientIp":"192.168.65.1"                   ← CORRECT
# "naiveClientIp":"1.2.3.4"                          ← still wrong, kept for comparison

The nginx config that produces those two behaviors differs by exactly one line:

location /api/ {
    proxy_pass http://api_pool/;
    # APPEND: keep what the client sent, add the real IP at the end.
    # Use this when there is ANOTHER trusted proxy in front of the gateway (a CDN,
    # say) and the app needs the whole chain. The app MUST read from the right.
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP       $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /safe-api/ {
    proxy_pass http://api_pool/;
    # OVERWRITE: wipe out everything the client sent. Use this when the gateway is
    # the FIRST hop touching the Internet. This is the safest default.
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Real-IP       $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
}

4.3 Path-based routing — and a measurement that goes against prediction

The most obvious job of a gateway: look at the path, decide where to send it. /api/orders/* to the orders service, /api/users/* to the users service, /static/* answered by the gateway itself. The client sees a single domain and has no idea how many services sit behind it.

"Let the gateway serve static files" — measure before you believe it

Classic advice: let the gateway serve static files directly, and every file it returns is one request the app doesn't have to handle. nginx reads files far faster than Node, so this sounds like it doesn't even need checking. I measured it anyway: 16 connections, 8 seconds, three repeats, comparing /static/hello.json (nginx serves it directly) against /api/fast (forwarded to the app):

The static file is 2.6x slower than proxying to a Node process. This is completely backwards from the prediction, so it's one of two things: either the advice is wrong, or I'm measuring something different from what I think I am.

Hypothesis: the static directory is bind-mounted from the host machine into the container. On Docker Desktop (macOS/Windows), a bind-mount crosses the VM boundary, so every syscall against that file is noticeably more expensive. And by default nginx reopens the file on every request: stat(), open(), read(). If the hypothesis is right, caching the metadata and the file descriptor should erase most of the gap. Four lines:

location /static/ {
    alias /usr/share/nginx/static/;
    expires 1h;

    # Cache metadata + file descriptor instead of stat()+open()+read() every request.
    open_file_cache          max=1000 inactive=60s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 1;
    open_file_cache_errors   on;
}

2.4x faster, p50 drops from 1.6 ms to 0.68 ms. The hypothesis holds: the bottleneck wasn't nginx, it was the filesystem underneath it. After the fix, the static route is nearly as fast as the proxy route — still a bit behind, and with a file larger than the lab's static/hello.json (94 bytes) the advantage would clearly tip toward nginx: the fixed cost per request (internal TCP handshake, event loop) makes up a bigger share when the file is tiny, while nginx reads a larger file straight from cache without spending any CPU on JavaScript.

Versioning the API

When an API has to change in a backward-incompatible way, the gateway is the natural place for two versions to coexist. Three common approaches:

In practice, path-based versioning wins in most systems for a very practical reason: when there's an incident at 2 a.m., you can read the version right off the log line without opening anything else.

The gateway is also the place to canary: push a small slice of traffic to the new version before cutting over completely. nginx does this with split_clients, and the important part is that it hashes on a stable key so one client always sees the same version:

# Hash on IP (or better: on the user id in a cookie) => ONE client always sees
# the SAME version. Hashing on $request_id instead makes the same person bounce
# between v1 and v2 across requests — a bug that's very hard to reproduce.
split_clients "${remote_addr}canary" $api_backend {
    5%      api_v2;
    *       api_v1;
}

location /api/ {
    proxy_pass http://$api_backend/;
}

5% here is a cautious starting point, not a fixed number — ramp it up (5% → 25% → 100%) as long as api_v2's separately measured error rate and latency stay healthy, and keep an immediate rollback to 0% ready in case they don't.

4.4 Cross-cutting concerns: what belongs on the gateway

There's a group of jobs every service needs: authentication, rate limiting, logging, compression, CORS. If every service implements its own, you get N implementations and N chances for one of them to get it wrong. The gateway is the place to do it once. But not everything belongs up there.

The boundary comes down to one question: does this task need to know the business? If not, gateway. If it does, service. Applying that question to every task is the only method I know of to keep the gateway from slowly turning into a place where logic accumulates.

Aggregating multiple services: latency equals the slowest branch

A single screen often needs data from several services. If the client calls each one itself, it eats the full network latency of doing so — expensive on a mobile network. Having the gateway (or a BFF service — backend for frontend) make those calls internally and return them once is a common design. The crucial part is calling them in parallel:

The lab's /aggregate endpoint really does call three branches over HTTP, so the socket and event-loop cost is real, not a fake setTimeout:

curl -sk "https://localhost:8443/api/aggregate?branches=50,120,200&mode=parallel"
# "maxBranchMs":200, "sumBranchMs":370, "totalMs":208.36

curl -sk "https://localhost:8443/api/aggregate?branches=50,120,200&mode=sequential"
# "maxBranchMs":200, "sumBranchMs":370, "totalMs":386.00

# Same three branches. Parallel: ~MAX. Sequential: ~SUM.

nginx comes with rate limiting built in — but you need to understand exactly how much bursting it allows, because that's where surprises usually happen:

# Shared bucket, keyed by the real IP (do NOT use raw XFF - see section 4.2).
# 10r/s is the REFILL rate, not "10 requests per second, max".
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

location /api/ {
    # burst=20: let up to 20 requests queue up, then drain them at 10r/s.
    # nodelay:  serve those 20 queued requests IMMEDIATELY instead of pacing them.
    # => with nodelay, a client can burst 20 requests in a fraction of a second.
    #    Without nodelay it is smoother, but the client sees the delay stretched out.
    limit_req zone=perip burst=20 nodelay;
    limit_req_status 429;          # default is 503 — 429 is more meaningful
    proxy_pass http://api_pool/;
}

4.5 The gateway as a single point of failure — and a timeout budget

Everything in this lesson funnels into one place, and that place sits on the path of 100% of the traffic. If the gateway dies, the whole system dies, even though every service behind it is perfectly healthy. The paradox: the component added to increase reliability is the most dangerous one.

Timeout budget

Every hop along the path has its own timeout, and they must shrink as you go further in. If an inner hop waits longer than an outer one, the excess wait is work that's guaranteed to be thrown away: the client is long gone, but the service is still holding a connection and CPU to compute a result nobody will receive.

Summary

Reverse proxy, load balancer and API gateway are three levels of responsibility at the same position, and each level adds power by adding state that has to be managed. TLS termination is cheap with persistent connections and many times more expensive without them — the cost lives in every connection, not every request. X-Forwarded-For has to be read from the right, or better yet, let the gateway overwrite it. Whatever doesn't need business knowledge belongs on the gateway; whatever does belongs on the service. And everything funneling through the gateway means the timeout budget and retry policy have to be designed on purpose, not left at their defaults.

The static-file measurement in this lesson leaves a lesson bigger than architecture itself: a number that contradicts your prediction is not a conclusion, it's a question. Had I stopped at the first measurement, this lesson would have stated "nginx serves static files slower than Node" — wrong, but backed by numbers, so it would have sounded thoroughly convincing.

Lesson 5 moves into the layer right after the gateway: caching. The lab already has Redis waiting in the cache profile, and the central question is the one every caching layer must answer — how stale can data get before it's unusable, and whose job is it to invalidate it.