Distributed Tracing
IntermediateDistributed tracing tracks a request as it flows through multiple microservices, creating a visual trace of the entire call chain with timing data. It enables pinpointing latency bottlenecks and failure points.
Overview
In a microservices architecture, a single user request (e.g. "place order") might flow through 10+ services. When something is slow or failing, you need to see the entire request journey. Distributed tracing instruments each service to propagate a trace context (trace ID + span ID) through HTTP headers or message metadata. Each service creates a span (a timed unit of work) and reports it to a tracing backend. The tracing system assembles spans into a trace — a tree-like structure showing the call chain with timing data for each hop. You can see that the /placeOrder request took 450ms total: 10ms in the API gateway, 200ms in the order service, 150ms in the payment service (of which 120ms was waiting for the bank API), and 90ms in the inventory service. Standards include OpenTelemetry (OTel), Jaeger, Zipkin, and AWS X-Ray.
Traces, Spans, and Context Propagation
A trace represents the entire request journey. Each service creates spans (timed operations). Context (traceId + spanId) is propagated via HTTP headers so all spans link together.
// Trace structure
//
// Trace ID: abc-123 (entire request)
//
// ┌─ Span 1: API Gateway (10ms) ─────────────────────────────────┐
// │ ┌─ Span 2: Order Service (200ms) ──────────────────────────┐│
// │ │ ┌─ Span 3: Payment Service (150ms) ───────────────────┐ ││
// │ │ │ ┌─ Span 4: Bank API call (120ms) ────────────────┐ │ ││
// │ │ │ └─────────────────────────────────────────────────┘ │ ││
// │ │ └─────────────────────────────────────────────────────┘ ││
// │ │ ┌─ Span 5: Inventory Service (90ms) ──────────────────┐ ││
// │ │ └─────────────────────────────────────────────────────┘ ││
// │ └──────────────────────────────────────────────────────────┘│
// └──────────────────────────────────────────────────────────────┘
// Total: 450ms — bottleneck is Bank API call (120ms)
// Context propagation via HTTP headers (W3C Trace Context)
// traceparent: 00-abc123def456-span789-01
// tracestate: vendor=value
// Spring Boot auto-propagation (Micrometer Tracing)
// Trace context automatically added to:
// - Outgoing HTTP requests (RestTemplate, WebClient, Feign)
// - Kafka messages (trace headers in Kafka record headers)
// - JDBC queries (as spans)OpenTelemetry & Jaeger
OpenTelemetry (OTel) is the industry standard for instrumentation. Jaeger and Zipkin are popular tracing backends that store, search, and visualise traces.
// OpenTelemetry + Jaeger setup (Spring Boot)
// build.gradle
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
// application.yml
management:
tracing:
sampling:
probability: 0.1 # sample 10% of requests (production)
otlp:
tracing:
endpoint: http://jaeger-collector:4318/v1/traces
// Custom span for business logic
@Service
public class PaymentService {
private final Tracer tracer;
public PaymentResult charge(PaymentRequest req) {
Span span = tracer.nextSpan().name("payment.charge").start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
span.tag("payment.amount", String.valueOf(req.getAmount()));
span.tag("payment.method", req.getMethod());
PaymentResult result = gateway.charge(req);
span.tag("payment.status", result.getStatus());
return result;
} finally {
span.end();
}
}
}
// Sampling strategies:
// 100% in dev/staging
// 1-10% in production (cost vs visibility trade-off)
// Always sample errors and slow requests (tail-based sampling)Key Points to Remember
- 1A trace tracks a request across multiple services; each service creates spans (timed operations).
- 2Context propagation (traceId + spanId) via HTTP headers links spans into a complete trace.
- 3OpenTelemetry (OTel) is the industry standard for instrumentation — vendor-neutral.
- 4Sample 1-10% of requests in production to balance cost and visibility; always sample errors.
- 5Tracing pinpoints latency bottlenecks: "the bank API call in payment service takes 120ms."
Interview Questions
Sign in to ask AriaWhat is distributed tracing and why is it needed?
How is trace context propagated between services?
What is sampling in distributed tracing and why is it necessary?
Compare Jaeger, Zipkin, and AWS X-Ray.
Design the observability stack for a 100-service microservices platform.
Ask Aria about Distributed Tracing
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.