01 The Problem

A checkout service's instances load a product catalog into memory on startup, about 25 seconds of work, before they can answer a request correctly. The single /health endpoint every instance exposes returns 200 the moment the process is up and listening, catalog loaded or not. During a routine scale-out, three new instances join the load balancer's pool, pass that health check instantly, and start receiving real traffic before the catalog finishes loading. For 25 seconds, a slice of real checkout requests hit a cold instance and come back with "product not found," not because anything is actually broken, but because the health check never asked whether the instance was ready, only whether the process had started.

The same single endpoint causes the opposite problem a few weeks later. Checkout gets a genuine traffic spike, instances are busy but working fine, and one instance's event loop is briefly backed up enough that its /health endpoint itself takes four seconds to respond instead of forty milliseconds. The orchestrator, using that same endpoint to decide whether to restart the pod, treats the slow response as a failed liveness check and kills it. The instance wasn't stuck. It was just busy, at the exact moment losing an instance hurts the most, and now there's one fewer instance handling a spike that was already straining the fleet.

Same root cause both times: one check was being asked to answer two different questions, and it can only ever give one answer.

02 What It Is

A liveness check answers one specific question: is this process alive at all, or is it stuck in a way it will never recover from on its own, a deadlock, an infinite loop, a state no amount of waiting fixes. Failing liveness has exactly one correct response: kill the process and start a fresh one, because whatever's wrong isn't going away by itself.

A readiness check answers a different question: can this specific instance handle traffic right now. A process can be completely alive, not stuck, not crashed, and still not ready, still loading a cache on startup, still finishing in-flight requests during a graceful shutdown, or genuinely too overloaded to take on more work this second. Failing readiness calls for a different response: stop routing new traffic here, but leave the process alone. Whatever it's doing will resolve on its own.

Conflate the two, using one check to answer both questions, and both failure directions become real. A process still warming its cache reports as merely "alive," with no separate readiness signal, and a load balancer that only checks liveness sends it real traffic before it's actually ready. Or the reverse: a process that's just busy, a garbage collection pause, a legitimate spike making the health endpoint itself slow to answer, fails what's being treated as a liveness check, and gets killed and restarted for being "dead" when it was only ever momentarily slow.

In Plain Terms

A new hire clocking in for their first shift is present, badge scanned, on the clock, definitely at their desk. That's not the same fact as being ready to take a live customer call, they might still be listening to training recordings for another hour. A phone system that only checks "clocked in" to decide who gets the next call routes a real customer to someone who's never taken one before. The reverse mistake is just as real: an experienced rep on a long, completely normal call isn't gone, they're just busy, and a system that can't tell busy from gone would yank their badge and send someone to check their desk over nothing.

03 How It Works

Try it: leave it on Combined, start the instance, and send traffic before it finishes warming up, the request fails even though the process is alive. Simulate momentary business and watch it get killed for being briefly slow. Switch to Separate liveness & readiness and do both again, traffic waits instead of failing, and a slow response no longer gets the instance killed.

04 Trade-offs

	One combined health check	Separate liveness and readiness checks
A cache still warming on startup	Reports 'healthy' if the process responds at all, traffic arrives before it's ready	Readiness fails independently, no traffic until warm, process itself untouched
A process under a legitimate load spike, briefly slow	Can look 'unhealthy' if the check times out, gets killed and restarted unnecessarily	Liveness still passes unless truly stuck, readiness can fail without triggering a restart
Graceful shutdown, draining in-flight requests	No clean way to say 'stop sending new traffic, but don't kill me yet'	Readiness fails on purpose during drain, liveness stays passing until the process actually exits
Implementation cost	One endpoint, simplest to wire up	Two endpoints, two different questions to actually answer correctly

05 Where You'll Actually See This

Load balancers named this gap explicitly in its own Watch Out: spreading load evenly and detecting failure fast are different problems. Worth being precise about which half of that this article closes. The polling-interval blind spot that Watch Out specifically named, a health check running every few seconds instead of continuously, is still real and still unresolved here. What this article answers is a different question underneath the same black box: what a health check is actually supposed to measure in the first place. A load balancer routing around "failing" instances is only as good as what "failing" means, and that article never unpacked it.

Circuit breaker is worth separating from this rather than treating as the same idea. A health check is external and periodic, something outside the process polling it on an interval, asking if it's okay whether or not anyone's currently sending it a request. A circuit breaker is internal and request-triggered, tracking failures as real calls actually happen and reacting only once enough of them fail. A service can pass every health check and still have a circuit breaker tripped in front of one specific downstream dependency, healthy overall, struggling with one specific call.

Neither case study implements explicit health checks, worth naming rather than glossing over. Both assume their single app server instance is either up or down, with no separate notion of warming up or draining. A reasonable simplification for a scaffold, but exactly the gap a real deployment behind a load balancer would need to close first.

Wiring up a separate readiness check is only half the work. Chaos engineering is the other half: actually degrading a real instance and watching whether the load balancer stops routing to it, rather than trusting that the endpoint returning the right status code in isolation means the same thing under a real failure.

How to reason about it

When asked how you'd set up health checks for a service, don't just say "add a health check endpoint." Say whether you mean liveness or readiness, because they answer different questions and trigger different actions: a failed liveness check should get a process killed and restarted, a failed readiness check should just stop new traffic from arriving. Then say how deep the readiness check actually goes, because a readiness check that pings every downstream dependency can turn one slow dependency into an outage for the entire fleet at once, instead of the graceful, partial failure health checks are supposed to buy you.

06 Quick Check