Observability

Advanced

Production & Advanced

Observability is knowing what your service is doing in production. The three pillars are metrics (Micrometer → Prometheus), logs (structured with MDC correlation IDs), and traces (Spring Micrometer Tracing → Zipkin/Grafana Tempo).

Overview

Micrometer is the metrics facade for Spring Boot — it instruments the JVM, Spring MVC, Spring Data, and more out of the box, and exports to Prometheus, Datadog, CloudWatch, or other backends. @Timed adds timing metrics to controller methods or services. Structured logging with MDC (Mapped Diagnostic Context) injects a correlation ID into every log line, making request tracing across log lines trivial. Spring Boot Actuator exposes all three signals via HTTP endpoints.

Micrometer Metrics — Custom Counters and Timers

Spring Boot auto-configures Micrometer with JVM, HTTP request, and data source metrics. Add custom metrics for business-level signals: payments processed, orders placed, Aria sessions started.

Java + YAML — Micrometer Counter, Timer, Prometheus export

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  metrics:
    tags:
      application: ${spring.application.name}  # tag all metrics

@Service
public class PaymentService {

    private final MeterRegistry meterRegistry;
    private final Counter paymentCounter;
    private final Timer paymentTimer;

    public PaymentService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;

        this.paymentCounter = Counter.builder("payments.processed")
            .description("Total payments processed")
            .tag("env", "production")
            .register(meterRegistry);

        this.paymentTimer = Timer.builder("payments.duration")
            .description("Payment processing latency")
            .register(meterRegistry);
    }

    public PaymentResult charge(ChargeRequest req) {
        return paymentTimer.recordCallable(() -> {
            PaymentResult result = processCharge(req);
            paymentCounter.increment(1, Tags.of("status", result.status()));
            return result;
        });
    }

    // Simpler: @Timed annotation (requires @EnableAspectJAutoProxy)
    @Timed(value = "payments.duration", description = "Payment charge latency")
    public PaymentResult chargeWithTimed(ChargeRequest req) {
        return processCharge(req);
    }
}

Structured Logging with MDC and Distributed Tracing

MDC (Mapped Diagnostic Context) attaches key-value pairs to every log line in the current thread. Inject a correlation ID at the start of every request so all log lines for a request can be found with a single query.

Java + YAML — MDC correlation ID filter + distributed tracing config

// ── MDC Correlation ID Filter ──────────────────────────────────────────────

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class CorrelationIdFilter extends OncePerRequestFilter {

    private static final String CORRELATION_ID = "correlationId";

    @Override
    protected void doFilterInternal(HttpServletRequest req,
                                    HttpServletResponse res,
                                    FilterChain chain) throws IOException, ServletException {
        String id = Optional.ofNullable(req.getHeader("X-Correlation-Id"))
            .orElse(UUID.randomUUID().toString().substring(0, 8));
        MDC.put(CORRELATION_ID, id);
        res.addHeader("X-Correlation-Id", id);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear(); // MUST clear after request — thread pool reuse
        }
    }
}

# application.yml — include MDC in log pattern
logging:
  pattern:
    console: "%d{HH:mm:ss} [%thread] %-5level [%X{correlationId}] %logger{36} - %msg%n"
  # Or use JSON structured logs for log aggregation (ELK, Loki)

# Output: 14:23:11 [http-nio-8080-exec-1] INFO  [a1b2c3d4] c.a.OrderService - Order ord_123 created

# ── Distributed Tracing (Spring Boot 3) ────────────────────────────────────
# pom.xml
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

# application.yml
management:
  tracing:
    sampling:
      probability: 0.1  # sample 10% of requests in production
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans
# Spring auto-injects traceId/spanId into MDC — visible in every log line
# and propagates W3C Trace Context headers to downstream Feign calls

Key Points to Remember

1Micrometer is the metrics facade — auto-configures JVM, HTTP, and data source metrics out of the box.
2@Timed adds latency metrics to methods; Counter tracks occurrence counts; Gauge tracks current values.
3Tag every metric with application name — required to filter metrics by service in Prometheus/Grafana.
4MDC injects key-value pairs (correlation ID) into every log line — clear it after each request with MDC.clear().
5Spring Boot 3 + micrometer-tracing auto-propagates trace/span IDs across service calls and into MDC.
6Sample 1–10% of traces in production with tracing.sampling.probability — full sampling is expensive.

Interview Questions

What is the difference between metrics, logs, and traces?

EasyAmazon

What is MDC and why must you call MDC.clear() after each request?

MediumThoughtWorks

How does Micrometer differ from Prometheus — what role does each play?

MediumAtlassian

What is a distributed trace and how does Spring propagate trace context across services?

HardNetflix

How would you alert on a payment failure rate exceeding 5% using Prometheus?

HardGoldman Sachs

Ask Aria about Observability

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Docker & Deployment