Distributed Tracing
IntermediateA trace ID propagates through all services involved in a request; spans capture latency at each hop, enabling end-to-end latency visualisation in Jaeger or Zipkin.
Overview
In a microservices system, a single user request may travel through dozens of services. When a request is slow or fails, you need to know which service is the culprit and how long each hop took. Distributed tracing solves this: a unique trace ID is generated at the entry point (API Gateway or first service) and propagated to every downstream service via HTTP headers (W3C TraceContext or B3). Each service creates a child span that records its own start time, end time, tags, and logs. All spans that share the same trace ID are collected and assembled into a trace — a tree of spans visualised as a timeline in Jaeger, Zipkin, or Tempo. In the Spring ecosystem, Micrometer Tracing (Boot 3+) replaced the older Spring Cloud Sleuth and integrates with both Brave (Zipkin) and OpenTelemetry.
Traces, Spans, and Context Propagation
A trace is the entire journey of a request across all services. A span is one unit of work within that trace — it records: - Service name and operation name - Start timestamp and duration - Parent span ID (to build the tree) - Tags (key-value metadata: http.method, db.statement) - Logs / events (timestamped annotations within the span)
Context propagation carries the trace ID and parent span ID between services in HTTP headers. The W3C TraceContext standard uses `traceparent` and `tracestate` headers. The older B3 format (Zipkin) uses `X-B3-TraceId`, `X-B3-SpanId`, and `X-B3-ParentSpanId`. Spring propagates these headers automatically when using RestTemplate, WebClient, or OpenFeign with Micrometer Tracing on the classpath.
# HTTP headers carrying trace context (W3C TraceContext format)
# traceparent: 00-{traceId}-{spanId}-{flags}
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
# B3 format (older, used by Zipkin/Spring Cloud Sleuth)
X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId: 00f067aa0ba902b7
X-B3-ParentSpanId: b7ad6b7169203331
X-B3-Sampled: 1
# Example trace tree for "Place Order" request:
# Trace: 4bf92f3...
# ├── Span: api-gateway [0ms → 250ms]
# │ ├── Span: order-service [5ms → 200ms]
# │ │ ├── Span: db-query[10ms → 40ms] ← SELECT
# │ │ └── Span: payment-service [50ms → 180ms] ← HTTP call
# │ │ └── Span: stripe-api [55ms → 175ms] ← externalSpring Boot 3 + Micrometer Tracing Setup
Spring Boot 3 (Micrometer Tracing) replaces Spring Cloud Sleuth. Add the micrometer-tracing bridge for your backend (Brave for Zipkin, or OTel for Jaeger/Tempo) and a reporter dependency. Tracing is then automatic for RestTemplate, WebClient, @KafkaListener, and more.
<!-- pom.xml — Spring Boot 3 with Micrometer Tracing + Zipkin -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-sender-urlconnection</artifactId>
</dependency>
# application.yml
management:
tracing:
sampling:
probability: 1.0 # 100% sampling (use 0.1 in production)
spring:
zipkin:
base-url: http://zipkin:9411
# For OpenTelemetry + Jaeger:
# management.otlp.tracing.endpoint: http://jaeger:4318/v1/traces
// Custom span in business code
@Service
@RequiredArgsConstructor
public class PaymentService {
private final Tracer tracer;
public void charge(String orderId, BigDecimal amount) {
Span span = tracer.nextSpan().name("charge-card").start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
span.tag("order.id", orderId);
span.tag("amount", amount.toString());
stripeClient.charge(amount); // external call
} finally {
span.end();
}
}
}Trace-Log Correlation — Linking Traces to Logs
Distributed tracing is most powerful when traces are correlated with logs. Micrometer Tracing automatically injects traceId and spanId into the MDC (Mapped Diagnostic Context), which means every log line printed during a traced request includes these IDs. In a centralised logging system (ELK / Loki), you can jump from a slow span in Jaeger directly to the matching log lines in Grafana.
# Logback pattern — include traceId and spanId in every log line
# logback-spring.xml
<pattern>
%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level [%X{traceId},%X{spanId}] %logger - %msg%n
</pattern>
# Example log output:
# 2024-03-15 10:23:45 [http-nio-8080-exec-1] INFO [4bf92f35,00f067aa] PaymentService - Charging order ORD-123
# In Grafana Loki — search by traceId to see ALL logs for a request:
# {app="order-service"} |= "4bf92f35"
# In Jaeger — click a slow span to open the trace ID in Loki/Kibana:
# This is the "trace-to-log" exemplar linkKey Points to Remember
- 1A trace = one request end-to-end; a span = one unit of work within that trace. Spans form a parent-child tree.
- 2Context propagation uses HTTP headers (W3C traceparent or B3 format) to carry trace ID across service boundaries.
- 3Spring Boot 3 uses Micrometer Tracing; Spring Boot 2 used Spring Cloud Sleuth — both instrument RestTemplate/WebClient/Feign automatically.
- 4Set sampling.probability=0.1 in production — 100% sampling is only for debugging; it creates excessive load on the tracing backend.
- 5Micrometer injects traceId/spanId into MDC automatically so all log lines for a request are correlatable.
- 6Standard observability trinity: logs (what happened), metrics (how often / how fast), traces (where time was spent).
Interview Questions
Sign in to ask AriaWhat is the difference between a trace and a span in distributed tracing?
How is a trace ID propagated between microservices?
What replaced Spring Cloud Sleuth in Spring Boot 3?
How would you correlate a slow Jaeger trace with the corresponding log lines in Kibana?
Why should you not use 100% sampling probability in production?
Ask Aria about Distributed Tracing
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.