01 The Problem

A checkout team finally ships distributed tracing after the kind of incident metrics, logs & traces describes, too many hours spent matching timestamps across six services by hand. A few weeks later, a slow checkout request gets flagged, and someone pulls up its trace to see where the time went. The trace has exactly two spans: API gateway, then pricing service. Nothing after that. To the engineer looking at it, the request appears to stop existing the moment it leaves pricing.

Except it didn't. The order was placed, the card was charged, the confirmation email went out. Payment definitely ran. It just isn't in the trace. An hour of digging turns up the actual cause: pricing recently switched to a new internal HTTP client for its outbound calls, and the new client doesn't forward the trace header by default, nobody wired it up. Pricing's call to payment goes out with no trace id attached at all. Payment, receiving a request with nothing to continue, does exactly what it's supposed to do when there's no trace id: starts a brand new one. Somewhere in the collector sits a second, completely disconnected trace, two spans of its own, payment and everything downstream of it, with no link back to the first one at all.

Nothing failed. No error was logged, no alert fired, the request succeeded end to end. The only symptom was a trace that looked shorter than it should have, and unless someone already knows that a dropped header produces exactly this shape of gap, it reads like a mystery instead of what it actually is: one specific call, in one specific service, that silently stopped forwarding a header.

02 What It Is

A trace id is generated exactly once, at the first point a request enters the system, usually the edge service or API gateway. From there, it has to travel with the request through every hop: every outbound call one service makes to another has to carry that same trace id along, typically as an HTTP header. Each service that receives it does two things. It creates its own span, a record of which operation ran, how long it took, and which span it descended from. And it forwards the same trace id on whatever outbound calls it makes next, so the next service can do the same thing.

Each service reports its finished spans to a collector, separately, after the fact, not as part of the request's own critical path. The collector isn't in the request's way, it just receives a stream of spans from every service, and reassembles them into one tree keyed by trace id whenever someone wants to look at a specific request's full path.

That reassembly only works if the trace id actually made it through every hop. Propagating the header is either something every engineer has to remember to do by hand on every outbound call, or something a shared library or middleware does automatically for whatever HTTP clients it wraps. Either way, the moment one hop doesn't forward the header, the chain breaks silently. The call itself still goes out, still gets a response, still succeeds. The receiving service just gets a request with no trace id, and has no way to know a trace was even in progress, so it starts a new one. From the outside, the original trace simply stops. Not because anything failed. Because nothing told the next hop to keep going.

In Plain Terms

A single package moves through a chain of couriers, each one required to scan the same tracking number before handing it to the next courier. The package itself never notices a missed scan, it still gets picked up, still gets carried, still arrives on time. But if one courier's scanner is broken, or they just forget, the tracking system shows the package's journey ending at that courier, with the rest of the trip either untracked entirely or picked up under a brand new, unrelated tracking number nobody thinks to connect back to the first one. Anyone checking the original number sees the story stop. Not because the package got lost. Because nobody scanned it forward.

03 How It Works

Rendering diagram…

04 Trade-offs

	Manual propagation	Shared middleware / instrumentation library
Who's responsible for forwarding the header	Every developer, on every outbound call, by hand	The framework or library forwards it automatically for calls it wraps
What happens with a new HTTP client or language	Silently drops the header unless someone remembers to add it	Breaks only if that specific client isn't covered by the instrumentation, a narrower gap, not a zero one
Failure mode	Common: one missed call anywhere in the chain breaks the trace, no error, just a gap	Rarer, but not zero: a library update, an uninstrumented dependency, a raw call outside the framework
Effort to adopt	Nothing to configure, but relies on every engineer remembering, indefinitely	Upfront cost to standardize on shared clients or an SDK, then mostly automatic

05 Where You'll Actually See This

Microservices architecture is what makes this mechanism necessary at all. A single process has one call stack, and a slow function is already visible in a profiler without any of this. The moment a request crosses several independently deployed services, there's no shared call stack left, and propagating a trace id by hand or by shared middleware is what has to replace it.

Metrics, logs & traces named a trace as the thing that answers where the time went across a whole request. This article is the actual mechanism underneath that claim, and specifically why it can fail quietly: the trace only stays connected as long as every hop keeps forwarding the id, and nothing forces that to happen.

Message queues is exactly where this propagation problem gets harder, not easier. A queue consumer has no HTTP request to read a header from at all, the trace id has to be written into the message's own metadata by the producer and read back out by the consumer, a genuinely different code path than the header-forwarding this article describes, and just as easy to forget.

Neither case study implements distributed tracing, worth naming rather than glossing over. Both are simple enough, few enough services in the request path, that a full trace would be overkill for what a couple of log greps already answer. The moment either grew into the six-service shape this article's own scenario describes, propagation is the exact piece that would need to be added deliberately, not something either design gets for free.

How to reason about it

When asked how you'd debug a slow request across several services, don't just say "we have distributed tracing." Say what actually has to happen for that to work: a trace id generated once, at the edge, carried forward on every outbound call a service makes, whether that's a shared library doing it automatically or a header someone remembered to add by hand. Then say what a broken trace actually looks like: not an error, just a gap, because that's the failure mode that shows up in production, and the one most engineers haven't been burned by until they have.

06 Quick Check