01 The Problem

A public API offers stock-price lookups to third-party trading dashboards, capped at 100 requests a minute per client. One integration partner ships a bug: their retry logic doesn't back off, it just hammers the endpoint in a tight loop on any non-200 response. Within minutes that one client is sending thousands of requests a second, and the server's connection pool is entirely consumed answering, or rejecting, that one client's flood. Every other client, who did nothing wrong, starts timing out too, because there's no capacity left for them.

Adding a limit fixes that: reset a per-client counter to zero at the top of every minute, allow up to 100 requests before it hits zero, reject the rest. It fixes the obvious case. It doesn't fix the client that happens to send 100 requests at 11:00:59, the last second of one window, and another 100 at 11:01:00, the first second of the next. Two hundred requests in about a second, and on paper, both windows are perfectly compliant, 100 requests apiece.

This is rate limiting's actual hard part. The limit itself is trivial to write down. Making it hold up against a burst that straddles the boundary between two counting windows is not.

02 What It Is

Rate limiting caps how many requests a client is allowed to make in some period, rejecting or delaying anything over that cap so no single client consumes more than its share of shared capacity. The mechanism enforcing it is what matters. A fixed window counts requests in discrete, non-overlapping blocks of time, resetting to zero at each boundary. A token bucket instead gives a client a bucket holding up to N tokens, refilled continuously rather than all at once, and each request consumes one token, allowed only when a token is available.

The fixed window's flaw isn't that it counts wrong, it counts exactly right within each window. The flaw is that "within each window" is the wrong boundary to reason about. A client can spend its full allowance in the last instant of one window and its full allowance again in the first instant of the next, and from the limiter's point of view both are separately valid, fully compliant 100-requests-per-minute windows. The server on the receiving end doesn't experience two compliant windows. It experiences one burst of 200 requests in about a second.

In Plain Terms

A parking garage that resets its "50 cars per hour" counter at the top of every hour can watch 50 cars pull in at 1:59 and another 50 pull in at 2:00, a hundred cars arriving within two minutes, and every single one of them was technically within that hour's limit. The garage didn't lie about its rate. It just measured the rate in blocks that don't reflect how close together two blocks' worth of cars can actually show up at the gate.

03 How It Works

Try it: leave the bucket full and click Send Request as fast as you can. Then use Wait 10 seconds to watch it refill without sitting through actual real time.

Rendering diagram…

04 Trade-offs

	Fixed window	Token bucket
Boundary behavior	Allows up to 2x the stated limit in a burst that straddles two windows	Bounded by bucket size regardless of when requests land, no boundary to straddle
Implementation	A counter and a reset timestamp, about as simple as rate limiting gets	Needs a token count and a refill rate tracked continuously, more moving parts
Burst tolerance	All-or-nothing per window: either you fit in this window's quota or you don't	Deliberately allows a burst up to the bucket's capacity, then throttles to the refill rate
Best for	Rough, cheap protection where the boundary edge case is an acceptable risk	Anywhere the boundary burst is a real problem, which is most public APIs

05 Where You'll Actually See This

Neither case study implements rate limiting explicitly, worth naming rather than glossing over. The URL shortener case study's POST /shorten is exactly the kind of public write endpoint that needs it in a real deployment, unlimited free code generation is an open invitation to abuse the base62 keyspace that case study's own Compare section spent time reasoning about.

It's also worth separating this from load balancers: both sit in the request path deciding what happens to traffic, but they're not solving the same problem. A load balancer distributes traffic across however much capacity exists. Rate limiting caps how much total demand is allowed to exist in the first place, per client, regardless of how much capacity is behind it. Perfectly distributed load across ten instances doesn't help if one client is generating enough traffic to need an eleventh. Rate limiting is what stops that traffic from being generated at all.

In practice this usually lives in an API gateway rather than in each backend service, the same consolidation argument that applies to auth applies here: one place enforcing the limit for every service behind it, instead of each service tracking its own counters and inevitably drifting.

Message queues covers a related but different case: a consumer calling a downstream API. That article's retry-with-backoff is reactive, back off after the downstream rejects a call. A consumer that already knows the downstream's limit can use the exact token bucket described above proactively instead, throttling its own outbound calls so it never hits the failure case that backoff exists to recover from.

Worth separating this from a circuit breaker too, since both cap how much a caller sends somewhere. Rate limiting is a policy decided in advance, this client gets this much, regardless of how the downstream is doing right now. A circuit breaker reacts to the downstream actually failing and stops calling it at all for a while. One shapes your own traffic on purpose. The other responds to someone else's health.

How to reason about it

When asked how you'd rate limit an API, don't stop at "track requests per client and reject over the limit." Say which algorithm and why: a fixed window if the boundary burst is genuinely an acceptable risk, a token bucket if it isn't. If you can't explain what happens to a client that sends its full quota at 11:00:59 and again at 11:01:00, you haven't chosen an algorithm, you've just described what rate limiting means in general.

06 Quick Check