01 What it is

Latency is the time a single request takes from sent to answered. Throughput is the number of requests a system completes per unit of time. They're different axes, and they don't move together: you can improve one while making the other worse. A layer that waits 50ms to collect requests into a batch before processing them adds up to 50ms to every individual request's latency, and increases throughput, because the fixed overhead of processing gets paid once per batch instead of once per request.

That's why the two get named separately instead of folded into one "performance" number. Optimizing for one without checking the other produces systems that are fast for a single user and fall over under concurrent load, or systems that handle enormous aggregate volume with individually painful response times.

02 Why it matters

The failure mode is specific: a request that takes 50ms when tested alone still takes 50ms to process under real traffic, but the wait before it starts processing balloons, because it's now queueing behind a fixed pool of workers or connections that can't drain requests as fast as they're arriving. Nothing got individually slower. Latency degrades as a symptom of hitting a throughput ceiling, not because any single operation changed. If the only thing you measured in staging was one request's response time, you'll ship something that looks fast right up until concurrent load exceeds capacity, and then every request behind the front of that queue pays for it.

In Plain Terms

A single ATM takes exactly 20 seconds to hand you your cash, every time, whether you're the only person there or the fiftieth person in line. That 20 seconds never changes. What changes is how long the fiftieth person waits before it's even their turn: if the machine can only serve one person at a time and fifty people show up in the same five minutes, that last person might wait fifteen minutes for a transaction that itself still only takes 20 seconds. The machine isn't slower. There are just more people than one machine can get through.

03 How it works

Rendering diagram…

The queue is doing the work of separating the two ideas visually. Processing time is fixed per worker. Wait time is what grows when arrival rate outpaces the workers' combined capacity, and wait time is invisible until you're already past the point where it matters.

04 Trade-offs

	Process immediately	Batch before processing
Per-request latency	Lowest possible: no wait beyond processing time	Adds the batch window's wait time on top of processing
Throughput	Limited by per-request overhead paid every single time	Higher: fixed overhead (a DB round trip, a lock, a network call) is amortized across the whole batch
Best for	Interactive requests with a human waiting on the response	Background or bulk work where nobody's watching a single request complete
Tail behavior under load	Degrades sharply once concurrency exceeds capacity, queueing shows up directly in latency	Degrades more gracefully; the batch window is a known, bounded cost regardless of load

05 Where you'll actually see this

The payment gateway case study's estimation section is this relationship in practice: 500 requests/second at an 800ms processor latency works out to roughly 400 requests in flight at once. That number, throughput times latency, is what actually sizes the connection pool, not a guess at "how many connections feels safe."

Raising that ceiling doesn't always mean adding workers. Caching raises it by removing repeat work from the queue entirely: a read a cache can answer is a read the fixed pool of workers never has to process at all.

Adding workers is still the other lever, and scalability covers it, but it's worth being precise about what it moves. Ten workers instead of one raise the ceiling this article is about, how many requests get processed per second, without doing anything for how long any single one of those requests takes once it's actually running. More throughput, not necessarily lower latency.

How to reason about it

If someone says p99 latency spiked under load, don't assume a single request got slower, ask whether requests are queueing behind a fixed pool of workers or connections instead. The fix for a throughput ceiling (add capacity, or shed load) is different from the fix for something actually taking longer to process (optimize the operation itself), and conflating the two means fixing the wrong problem.

06 Quick Check