01 The Problem

A webinar platform's app tier is stateless, sessions live in Redis, nothing important sits in one process's memory, so running ten instances instead of one should be trivial. It is trivial to run ten instances. What isn't trivial: a client only knows one hostname, so something in front of those ten instances has to receive every request and decide, per request, which instance actually handles it.

Two ways that decision goes wrong, both real. Round robin sends the first request to instance 1, the second to instance 2, on down the line, evenly, in order, regardless of what each instance is currently doing. During a big webinar, a handful of clients open long-lived signaling connections that each hold an instance busy for the length of the call, and round robin keeps handing that instance new requests anyway, unaware it's already loaded, while an instance two slots over sits nearly idle. Separately, mid-request, one instance starts failing its health check, maybe it's out of memory, maybe a deploy is rolling through it, and a request already in flight to it either gets retried somewhere else immediately or the client just gets a dropped connection. Nothing about "add a load balancer" answers which of those two happens by default.

This is what a load balancer actually has to get right: not just spreading requests around, but which algorithm fits how your requests behave, and what happens the moment one of the machines it's routing to stops answering.

02 What It Is

A load balancer sits in front of a pool of identical, interchangeable instances and decides, for every incoming request or connection, which instance handles it. It does that with a routing algorithm, the rule for picking a target, and a health check, a way to notice an instance has stopped answering and stop sending it traffic. Round robin cycles through instances in order. Least-connections sends each request to whichever instance currently has the fewest open requests. Consistent hashing routes based on a hash of something in the request, a client ID or session key, so the same client tends to land on the same instance. Each optimizes for a different shape of traffic.

Choosing the load balancer, the product or the software, is the easy part. The routing algorithm is the real engineering decision, because it encodes an assumption about your traffic: that every request costs about the same (round robin's assumption, wrong for the webinar's long-lived connections), or that current open-connection count is the best signal of capacity (least-connections), or that the same client landing on the same instance matters more than perfectly even load (consistent hashing). Pick the algorithm that matches an assumption your traffic doesn't meet, and you've rebuilt the exact hot spot the extra instances were supposed to fix.

In Plain Terms

A restaurant host seats parties as they arrive by walking down the row of tables in order, one, two, three, one, two, three, whether or not table two still has four people lingering over dessert. A host who actually looks at which tables are free before seating the next party solves that, but only if she checks often enough, and a brand-new table that just got wiped down looks empty even in the ninety seconds before someone finishes carrying dishes to it, so it can get seated with a party before it's actually ready.

03 How It Works

Rendering diagram…

Try it: switch to Least Connections, mark a server down, send a few requests to the remaining three, then mark it back up and send one more.

04 Trade-offs

	Round robin	Least-connections
What it optimizes for	Even request count across instances	Even current load across instances
Assumption it makes	Every request costs about the same to handle	Current open-connection count is a good proxy for how busy an instance actually is
Where it breaks	Long-lived or expensive requests pile up on one instance while others go idle	A brand-new instance with zero connections looks like the least busy one and can get flooded the instant it joins, before its real capacity is proven
Simplicity	Trivial to implement and reason about	Needs the load balancer to actively track live connection counts per instance

05 Where You'll Actually See This

The URL shortener case study's app server was already established as stateless in scalability, specifically so more instances could be added without a rewrite. A load balancer is what makes that actually work: it's the thing sitting in front of those instances deciding which one gets each request, the piece scalability's own argument assumes exists but never draws.

CDN is this same routing question at a different scale: a load balancer picks the best instance within one data center, a CDN picks the best edge location on the planet for a given user. Same underlying decision, which nearby, capable node should actually answer this, just applied across machines in one rack versus across the globe.

Rate limiting sits in the same request path but isn't answering the same question. A load balancer distributes however much traffic shows up across however much capacity exists. It has no opinion on whether that traffic should exist in the first place. Rate limiting is the thing capping total demand per client, and neither one substitutes for the other, perfectly distributed load doesn't help if one client is generating enough of it to need capacity that isn't there.

API gateways sit near the same point in the path too, and it's worth being precise about the difference rather than treating the two as interchangeable. A load balancer distributes requests across identical instances of one service. A gateway decides which different service a request belongs to in the first place, on top of auth, rate limiting, and logging along the way. A real system usually has both, layered, not one standing in for the other.

Both of those take it as given that there's more than one service to route to. Microservices architecture is the article that actually argues for that split, and what it costs to get there.

The Watch Out above named a real gap: spreading load and detecting failure fast are different problems. Health checks & readiness probes closes half of it, not the polling-interval blind spot itself, that's still real, but what "failing" is actually supposed to mean. A health check treated as one undifferentiated pass-or-fail signal is exactly how a warming-up instance ends up looking identical to a genuinely broken one.

How to reason about it

When asked how a load balancer works, don't stop at "it spreads traffic across instances." Name the algorithm and say why it fits the traffic: round robin for uniform, cheap requests, least-connections for requests that vary in cost, consistent hashing when the same client needs to keep landing on the same instance. Then say what happens to a request in flight when a health check fails, because that's the part people forget to design for until it happens in production.

06 Quick Check