01 The Problem

A checkout service has done the work: a circuit breaker in front of the fraud-detection service, a health check that's supposed to pull a struggling instance out of rotation, and a page configured to fire the moment the breaker trips open. The team has walked through the design in three separate reviews. Everyone is confident the system is resilient, in the specific sense that matters: if fraud-detection has a bad day, checkout isn't supposed to have one too.

Then fraud-detection has a bad day. Every call to it starts timing out at the full ten-second limit instead of failing fast. Twenty minutes in, checkout is down too, exactly the outcome the circuit breaker existed to prevent. Nobody gets paged, because the breaker that was supposed to trip and fire that page never trips at all. The postmortem finds the actual bug: the breaker's failure threshold was written as a percentage of total checkout traffic, not a percentage of calls to fraud-detection specifically. Fraud-detection calls are a small slice of everything checkout does, so even a 100% failure rate on fraud-detection alone never moves the number enough to cross a threshold measured against all of it. The breaker had been sitting there, correctly deployed, permanently unable to trip, since the day it shipped.

Nobody wrote bad code on purpose. The configuration read correctly in every review anyone gave it. It had just never been checked against a real failure, because nobody had ever actually made fraud-detection fail on purpose to see what the breaker would do about it.

02 What It Is

Chaos engineering is deliberately injecting a real failure into a system, on purpose, in a controlled way, specifically to find out whether the mechanisms meant to handle that failure actually do. A circuit breaker, a health check, an alert, a failover plan: all of these are code, configuration, or both, and code that has never run under the condition it was written for is a claim about what it does, not a fact.

The mechanism itself is narrow and deliberate. Pick one specific failure, a dependency timing out, an instance getting killed, a whole region losing network access. Inject exactly that failure into a real environment, usually with someone watching and a way to stop the experiment if it goes further than expected. Then observe whether the resilience mechanism that's supposed to react actually reacts the way everyone assumed it would. The breaker should trip. The health check should fail and pull the instance. The page should fire. If any of those doesn't happen, the experiment just found a real gap, on a schedule the team chose, instead of during a live incident nobody chose.

The point isn't chaos for its own sake, and it isn't about proving systems are fragile. It's that a resilience mechanism sitting unused during normal operation gives you zero evidence about whether it actually works. A breaker that's never tripped, a health check that's never failed, an alert that's never fired: none of them have been proven correct, only assumed correct, and assumed and proven look identical right up until a real failure tells them apart.

In Plain Terms

A building has a sprinkler system, installed, inspected on paper, signed off by whoever signs off on that kind of thing. Nobody has ever actually set off a real sprinkler head to check whether water comes out, because doing that means flooding part of the building on purpose. Every document describing the system says it's fine. Then there's an actual fire, and the water main feeding that floor turns out to have been shut off for a renovation eight months earlier and never reopened. The sprinkler system was never broken on paper. It was broken in the one way paper can't check, and the fire was the first real test it ever got.

03 How It Works

Rendering diagram…

04 Trade-offs

	Never tested in production	Tested via a chaos experiment
Whether the circuit breaker actually trips	Assumed correct because the configuration reads correctly	Proven by injecting a real failure against the actual dependency and watching it trip
Whether the alert actually fires	Assumed because the alert rule exists in a dashboard	Proven by watching a real page arrive during the experiment, not just checking the rule exists
Whether the health check actually removes a bad instance	Assumed because the endpoint returns the right status in isolation	Proven by actually degrading one instance and watching the load balancer stop routing to it
What a real incident finds	The first real failure is also the first real test, at the worst possible time	The gaps get found during a scheduled window with someone watching, not during a live outage

05 Where You'll Actually See This

Circuit breaker is the direct subject of this article's own scenario, worth being specific about what a chaos experiment actually proves about one. It isn't just confirming a breaker exists in the code. It's confirming the threshold trips against the traffic it's actually supposed to be measured against, that the cooldown is long enough for a real recovery, and that the half-open probe doesn't immediately reopen the circuit under real load, three separate things a code review can miss because none of them are visible by reading the configuration, only by watching it react to a real failure.

Health checks & readiness probes is the other mechanism this article's exercises point at directly. A readiness check that's supposed to pull a struggling instance out of rotation has exactly the same problem a circuit breaker does: it can be wired up, return the right status code in isolation, and still never once get exercised by a real failure until a chaos experiment actually degrades a real instance and watches whether the load balancer stops routing to it, not just whether the endpoint answers correctly on a normal request.

Disaster recovery's own warning, that a backup nobody's ever actually restored from is a claim and not a fact, is this same idea one level up. A documented failover plan that's never actually been run is in the same unproven state as a circuit breaker that's never tripped: it reads correctly, nobody's lying about it, and it has exactly zero evidence behind it. Actually failing over to the standby region on a schedule, watching RTO turn into a real, measured number instead of an assumed one, is a chaos experiment in every sense this article means, just aimed at a bigger blast radius.

Alerting & on-call is worth checking too, though it's a smaller piece of the same exercise rather than a separate prerequisite. A good chaos experiment doesn't stop at confirming the breaker tripped, it confirms the page that's supposed to fire the moment it trips actually arrives, on a real phone, not just that the alert rule exists somewhere in a dashboard. An alert that's never fired for real is exactly as unproven as a breaker that's never tripped, for exactly the same reason.

Neither case study has a circuit breaker, a health check, or an alert defined yet, worth naming rather than glossing over. Chaos engineering has nothing to test until one of those mechanisms actually exists. The moment either design added a circuit breaker in front of a real dependency or a health check behind its load balancer, proving those actually work under a real failure would be exactly this article's exercise, not before.

How to reason about it

When asked how you'd know a resilience mechanism actually works, don't just say "we have a circuit breaker" or "we have alerting." Say how you know, specifically whether anyone has ever actually triggered the failure it's supposed to handle and watched it respond correctly. An answer that stops at "it's configured" hasn't said whether it works, only that it exists. Those are different claims, and the gap between them is exactly where the outage in this article's own scenario came from.

06 Quick Check