How Distributed Tracing Works
IntermediateWhen one user request flows through a dozen microservices, a single log file cannot tell you where the time went or where it failed. Distributed tracing stitches the whole journey together. Each request gets a unique trace ID, and every service records a span — a timed unit of work — tagged with that trace ID and its parent. A tracing backend assembles the spans into a tree, so you can see the end-to-end path, latency of each hop, and exactly where errors occur.
Think of a tracking number for a parcel
A parcel passing through many depots carries one tracking number. Each depot scans it on arrival and departure, logging a timestamped event under that number. At the end you can see the parcel entire journey — which depot it sat in for hours, where it got stuck. A trace ID is the tracking number for a request; each service span is a scan, and the tracing tool reconstructs the full timeline across all the depots (services).
Step by Step
Key Concepts
Trace and Span
A trace is the whole journey of one request; a span is a single timed unit of work within it. Spans link to their parent to form a tree, and all share the trace ID.
Context Propagation
Passing the trace and parent span IDs across service boundaries (via headers like W3C traceparent) so each service joins the same trace rather than starting a new one.
OpenTelemetry
The vendor-neutral standard and SDKs for generating traces, metrics, and logs. It lets you instrument once and export to any backend (Jaeger, Zipkin, or a commercial tool).
Sampling
Recording only a fraction of traces to control overhead and cost at high volume. Head or tail sampling decides which requests to keep — often prioritising errors and slow requests.
Key Facts
- Tracing answers "where did the time go?" across services — something per-service logs and metrics alone cannot, because they lack the end-to-end request context.
- OpenTelemetry has become the de facto standard, so you can instrument once and switch backends without re-instrumenting.
- At scale you sample traces (keeping a percentage, plus all errors and slow requests) to keep overhead and storage costs manageable.
Real-World Applications
Finding the slow hop
A checkout that feels slow shows, in its trace, that one downstream inventory call takes 800ms while everything else is fast — pinpointing exactly which service to optimise.
Debugging a failure across services
When a request errors, the trace reveals which service threw the exception and the full chain of calls that led there, replacing hours of grepping disconnected logs.
Frequently Asked Questions
What is the difference between a trace and a span?
A trace represents the entire end-to-end journey of a single request through the system. A span is one timed unit of work within that trace — such as an HTTP handler or a database query — with a start and end time. Spans link to their parent span, and all spans in a request share the same trace ID, forming a tree that represents the trace.
How does trace context propagate between services?
When one service calls another, it includes the current trace ID and parent span ID in the outgoing request headers — commonly using the W3C traceparent header. The receiving service reads those headers and creates its spans as children of the same trace, so the whole request stays connected across service boundaries instead of appearing as separate, unrelated traces.
What is OpenTelemetry?
OpenTelemetry is a vendor-neutral, open standard and set of SDKs for generating and exporting telemetry — traces, metrics, and logs. You instrument your services once with OpenTelemetry and can export to any compatible backend such as Jaeger, Zipkin, or a commercial observability platform, avoiding lock-in to a single vendor tool.
Why is sampling used in distributed tracing?
Capturing and storing a trace for every single request at high volume is expensive in overhead and storage. Sampling records only a fraction of traces to keep costs manageable. Strategies like tail sampling can prioritise the most useful traces — for example, always keeping traces that contain errors or exceed a latency threshold.