01 The Problem

A checkout service deploys, and forty alerts fire inside two minutes: CPU spiked during the rollout, three pods restarted, memory crossed 70%, the connection pool briefly saturated, deploy duration ran longer than average. Every single one is true. Every single one is also exactly what happens on every deploy this team has ever shipped, expected, self-correcting, gone within a minute of the rollout finishing. The on-call engineer swipes the notifications away without reading them, the same way they do every deploy, because reading forty alerts that never mean anything stopped being worth the time months ago.

A few weeks later, a deploy ships a bad config, and checkout's actual success rate drops to 92%. Real users are getting real errors on a real purchase flow. If an alert exists for that at all, it's page forty-one in a wall of forty that already trained the on-call to swipe first and check later, and it gets dismissed with the rest, unread, the same reflex that's been correct forty times in a row. The outage runs for the better part of an hour before a support ticket surfaces it, not the alerting system that was supposedly watching this exact thing.

Nothing about those forty alerts was wrong. They were all true. That's the actual problem: true and worth waking someone up for are not the same claim, and a system that can't tell them apart eventually gets ignored uniformly, the one page that mattered included.

02 What It Is

A cause-based alert fires on the internal mechanism of the system: CPU utilization, memory, disk space, a pod restarting, queue depth. These are real, measurable facts, and they're also things that move constantly as ordinary consequences of autoscaling, deploys, and everyday traffic variation. Most of the time, a cause-level number moving doesn't mean anything is actually wrong for a user, it means the system is doing exactly what it's supposed to do under normal load.

A symptom-based alert fires on what a user actually experiences: the error rate users are hitting, the latency they're waiting through, whether an SLI has breached its SLO. These move far less often, because they only move when something genuinely affects someone on the other end of the request. A symptom-based alert staying quiet is a fact about whether things are fine. A cause-based alert staying quiet usually just means nobody's looking at the dashboard right now.

The actual design principle underneath both: a page should be actionable, something a human needs to do, right now, that an automated system can't do for them. CPU crossing 80% during a scale-out event isn't actionable, the system is already handling it. Checkout's success rate dropping below the SLO is actionable, because a human deciding to roll back, or escalate, or dig into the cause, is exactly what needs to happen next. Alerting on causes produces a flood of technically-true, rarely-actionable noise. Alerting on symptoms produces a small number of pages that are almost always worth answering, which is the only way a page stays trustworthy over months of on-call rotations instead of becoming background noise by the second week.

In Plain Terms

A smoke detector that goes off every time someone makes toast eventually gets its battery pulled. Nobody removes it because they don't care about fires, they remove it because it's been wrong, or at least irrelevant, so many times in a row that the sound stopped meaning anything. The detector going off for burnt toast and the detector going off for an actual fire are the exact same sound, and once a household has learned to ignore that sound on reflex, the fire doesn't get a different, more urgent noise to cut through the habit. It gets ignored right along with the toast.

03 How It Works

Rendering diagram…

04 Trade-offs

	Cause-based alerts	Symptom-based alerts
What triggers it	Internal system state: CPU, memory, pod restarts, disk usage	User-facing impact: error rate, latency, an SLO actually breached
Fires during a normal deploy	Constantly, these fluctuate as part of routine operation	Rarely, only if the deploy actually breaks something users notice
Signal quality over time	Degrades fast, gets muted or ignored once it's fired enough times for nothing	Stays trustworthy, because it only fires when something real is happening
What it's still good for	Debugging and dashboards, after something's already known to be wrong	Deciding whether to wake a human up right now

05 Where You'll Actually See This

SLIs, SLOs & error budgets is what a symptom-based alert is usually built on directly, the same specific, user-facing indicator that defines the budget in the first place. A well-designed alert here usually isn't just "the SLI just breached the SLO," reacting only once the budget's already spent. It's built on burn rate, how fast the error budget is being consumed right now, so a page can fire while there's still budget left to react within, not as an epitaph for one that's already gone.

Metrics, logs & traces is what an alert is almost always built on top of, specifically a metric, since a log line or a trace span isn't naturally an aggregated number you can set a threshold against. This article is what makes a metric worth alerting on in the first place, not just worth collecting. Once a page actually fires, though, distributed tracing is usually the next tool reached for: a metric decided something was wrong, a trace is what shows which specific hop in the request actually caused it.

A tripped circuit breaker is a genuinely good candidate for a page, not because CPU or memory moved, but because it means real requests are actually failing against a real dependency, right now. Worth naming that it isn't quite the same claim as an SLI breach, though: a circuit can trip on a dependency that's a small slice of overall traffic, worth waking up whoever owns that specific dependency, without necessarily meaning the whole service's SLO is anywhere near danger.

Neither case study defines any alerts explicitly, worth naming rather than glossing over. Both are simple enough that a human watching a single dashboard during business hours would probably catch a real problem. Neither currently distinguishes what would actually need to wake someone up at 3am from what's just worth knowing about later, exactly the line this article is about drawing.

How to reason about it

When asked how you'd design alerting for a service, don't just say "alert when something looks wrong." Say whether the alert is triggered by a cause, an internal system metric, or a symptom, something a user actually felt, and say why that distinction determines whether the alert stays trustworthy over time or gets muted after the first few false alarms. An answer that can't tell the difference between "CPU is high" and "users are getting errors" as reasons to page someone hasn't designed an alerting strategy, just a list of things that can be measured.

06 Quick Check