01 The Problem

A checkout request at an e-commerce company passes through six services: API gateway, auth, inventory, pricing, payment, notification. Checkout p99 latency alarms, jumping from 200ms to 4 seconds fifteen minutes ago. That's the entire content of the alert. Something, somewhere, in one of six services, got slow for some requests, during some window.

The on-call engineer opens the dashboard for each service. Payment's error rate ticked up in the same window. Grepping payment's logs around that timestamp turns up a line: "timeout calling card processor, retrying." That looks like the answer. Payment is slow, payment is the bottleneck, go look at payment's database queries.

Except the same fifteen-minute window had thousands of checkout requests, and most of them succeeded fine. Which specific requests were the slow ones the metric is complaining about. Was payment actually where the time went, or did payment's own call time out because inventory or pricing had already eaten most of the request's time budget upstream, leaving payment no room to breathe and a timeout that's a symptom, not a cause. The metric doesn't know, it only knows an aggregate number moved. The log doesn't know either, it knows that at some point, payment logged a timeout, for some request, with nothing tying that specific log line to the specific slow requests the metric is alarming about.

02 What It Is

Metrics are numbers aggregated over time: request count, error rate, latency percentiles, CPU usage. They're cheap to store and fast to query precisely because they throw away the detail of any single request the moment they're recorded. A p99 latency metric knows that some request, somewhere, took four seconds. It doesn't know which one, or why.

Logs are discrete events, usually one line per thing that happened, timestamped and written by whichever service produced them. A log line is detailed about the one thing it recorded, but it's an island. Nothing connects a log line in the payment service to a log line in the inventory service unless something explicitly ties them together.

Traces are the thing built to do that tying. A trace id gets generated once, at the edge, the moment a request first arrives, and travels with the request through every service it touches. Each service records a span: how long it spent, tagged with that same trace id. Collect every span sharing one trace id and you get the actual timeline of one specific request across every service it passed through, in order, with exact durations.

The three aren't competing options. They answer genuinely different questions, and none of them substitutes for the other two. A metric tells you something is wrong, in aggregate, right now. A log tells you what happened, at one specific point, in one specific service. A trace tells you where the time actually went, across the entire path one specific request took. A team with excellent metrics and excellent logging can still have zero ability to answer "which of these six services actually caused this one slow request," because that specific question needs correlation across services, and neither metrics nor logs provide it on their own.

In Plain Terms

A patient's vitals monitor beeping an alarm tells the nurse something is wrong, right now, in aggregate: heart rate spiked. A single note in the chart, "patient had a reaction at 3:15pm," tells you what happened at exactly one moment, but nothing about what led up to it. Only the full procedure record, the complete timeline of every drug administered and every reading taken across the entire three-hour surgery, lets someone actually line up the vitals spike with the specific drug given four minutes earlier. The monitor and the note were both accurate. Neither one, on its own, could show which moment in a three-hour procedure actually caused the spike.

03 How It Works

Rendering diagram…

Rendering diagram…

04 Trade-offs

	Metrics + logs, uncorrelated	Metrics + logs + traces, correlated by request
Tells you something's wrong	Yes, a metric alarms on the aggregate: p99, error rate	Yes, same alarm still fires
Tells you what happened at one point	Yes, a log line names an error and a timestamp	Yes, same log line, now tagged with a trace id
Tells you which service actually cost the time	No, you're matching timestamps across six services' separate log files by hand	Yes, the trace shows which span in the request dominates total duration
What it costs to get	Nothing extra, most systems already collect this	A trace id generated at the edge and propagated through every hop, header by header

05 Where You'll Actually See This

Microservices architecture is what actually makes this problem exist in the first place. A single monolithic process has one log file and one process's own metrics, there's nowhere else the time could have gone, and a stack trace already shows exactly which function was slow. The moment a request crosses six separate services, each with its own logs and its own metrics, that stack trace stops existing, and distributed tracing is the thing built to replace it.

Circuit breaker is exactly the kind of event these three signals need to explain together, not separately. A circuit tripping open is itself a metric, the open or closed state, the failure count that crossed the threshold. It produces a log line at the moment it trips. And if a trace happens to be running through a request that hit the open circuit, the trace shows exactly where the fast-fail happened instead of the timeout the caller would otherwise have waited out. Without a trace, you get a log saying a breaker tripped and a metric saying request volume to that dependency dropped, with no way to see whether that's the same incident as the latency spike already paged out on.

Message queues add a real wrinkle traces don't automatically solve. A trace id propagated through synchronous calls, one HTTP header per hop, doesn't cross a queue boundary for free. A consumer picking up a message has to read the trace id back out of the message itself and continue the same trace, or the trace simply stops at the producer and a disconnected one starts at the consumer, and the correlation this article is about breaks exactly at the async boundary.

Neither case study implements any of this, worth naming rather than glossing over. Both are simple enough, few enough services in the request path, that grepping one log file and eyeballing one dashboard actually is enough to diagnose a problem. The moment either design grew past a handful of services, the payment gateway's own retry logic, a call that might retry, might time out, might hit a downstream that's slow for reasons upstream doesn't know about, is exactly the kind of cross-service call that becomes genuinely hard to diagnose without a trace tying the whole thing together.

A metric collected here is also the raw material SLIs, SLOs & error budgets is built out of. An SLI is usually one specific metric, request success rate, latency under some threshold, chosen because it represents what a user actually experiences rather than whatever was easiest to already have a dashboard for. That article is what happens once a team has picked one of these numbers on purpose and decided how much of it they're allowed to lose before they've broken a promise, a decision this article doesn't make on its own.

Collecting a metric and deciding it's worth waking someone up over are two different decisions. Alerting & on-call is that second decision specifically, and it leans on a distinction this article doesn't make on its own: a metric moving is not the same claim as a metric moving in a way a user actually felt.

How to reason about it

When asked how you'd debug a distributed latency spike, don't just say "check the logs and the dashboards." Name what each signal can and can't tell you: a metric says something is wrong in aggregate, a log says what happened at one specific point, and only a trace ties one specific request's full path together so you can see which of several services actually cost the time. An answer that stops at "we have logging and monitoring" hasn't explained how you'd find which of six services caused one specific slow request. That's what tracing is for, and it's usually the piece teams add last, after they've already been burned by not having it.

06 Quick Check