Cheat SheetsMicroservicesObservability

Observability — Cheat Sheet

Microservices · 4 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org
Observability
Microservices4 topicsQuick revision reference
1

Distributed Tracing

A trace ID propagates through all services involved in a request; spans capture latency at each hop, enabling end-to-end latency visualisation in Jaeger or Zipkin.

  • ✓A trace = one request end-to-end; a span = one unit of work within that trace. Spans form a parent-child tree.
  • ✓Context propagation uses HTTP headers (W3C traceparent or B3 format) to carry trace ID across service boundaries.
  • ✓Spring Boot 3 uses Micrometer Tracing; Spring Boot 2 used Spring Cloud Sleuth — both instrument RestTemplate/WebClient/Feign automatically.
  • ✓Set sampling.probability=0.1 in production — 100% sampling is only for debugging; it creates excessive load on the tracing backend.
  • ✓Micrometer injects traceId/spanId into MDC automatically so all log lines for a request are correlatable.
  • ✓Standard observability trinity: logs (what happened), metrics (how often / how fast), traces (where time was spent).
HTTP Headers — Trace Context
# HTTP headers carrying trace context (W3C TraceContext format)
# traceparent: 00-{traceId}-{spanId}-{flags}
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

# B3 format (older, used by Zipkin/Spring Cloud Sleuth)
X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId: 00f067aa0ba902b7
X-B3-ParentSpanId: b7ad6b7169203331
X-B3-Sampled: 1

# Example trace tree for "Place Order" request:
# Trace: 4bf92f3...
#   ├── Span: api-gateway         [0ms → 250ms]
#   │     ├── Span: order-service [5ms → 200ms]
#   │     │     ├── Span: db-query[10ms → 40ms]  ← SELECT
#   │     │     └── Span: payment-service [50ms → 180ms]  ← HTTP call
#   │     │           └── Span: stripe-api [55ms → 175ms] ← external
2

Centralised Logging (ELK/Loki)

All services ship structured JSON logs to an aggregator (Elasticsearch, Loki); correlation IDs link logs across service boundaries for end-to-end debugging.

  • ✓Always emit structured JSON — free-text logs cannot be reliably parsed or indexed.
  • ✓Correlation IDs must be generated at the edge (API gateway) and propagated through every HTTP call and message header.
  • ✓Use MDC.put() + MDC.clear() in a filter so all log statements in a request automatically include the ID without explicit passing.
  • ✓Logstash/Loki labels should be low-cardinality (service, env, level); correlation ID belongs in the log body, not as a label.
  • ✓Log level discipline: ERROR for actionable failures, WARN for degraded state, INFO for business events, DEBUG for dev only.
  • ✓Use log sampling at high QPS to control ingestion costs — only sample DEBUG and INFO, never sample ERROR.
Java — Logback JSON + MDC correlation filter
<!-- pom.xml -->
<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>7.4</version>
</dependency>

<!-- logback-spring.xml -->
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>correlationId</includeMdcKeyName>
    <includeMdcKeyName>userId</includeMdcKeyName>
  </encoder>
</appender>

// CorrelationFilter.java
@Component
@Order(1)
public class CorrelationFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req,
            HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String id = Optional.ofNullable(req.getHeader("X-Correlation-Id"))
                            .orElse(UUID.randomUUID().toString());
        MDC.put("correlationId", id);
        res.setHeader("X-Correlation-Id", id);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}
3

Health Checks & Readiness Probes

Liveness probes detect deadlocked processes for restart; readiness probes tell the load balancer when a newly started instance is ready to accept traffic.

  • ✓Liveness → restart pod on failure; Readiness → remove from LB endpoints on failure (pod stays running).
  • ✓Never fail liveness for a slow DB — that causes a restart storm. Fail readiness for temporarily degraded dependencies.
  • ✓Startup probe protects slow-starting JVMs: liveness and readiness only activate after the startup probe succeeds.
  • ✓Spring Boot 2.3+ exposes /actuator/health/liveness and /actuator/health/readiness automatically with livenessState and readinessState enabled.
  • ✓Programmatically publish AvailabilityChangeEvent<ReadinessState> to dynamically toggle readiness based on application logic.
  • ✓Set terminationGracePeriodSeconds > Spring's timeout-per-shutdown-phase and add a preStop sleep to allow the LB to drain before Spring stops accepting requests.
YAML — Kubernetes Health Probes
# Kubernetes Deployment — health probe configuration
spec:
  containers:
  - name: order-service
    image: myregistry/order-service:1.2.3

    # Startup probe — runs first; liveness/readiness only start after this succeeds
    startupProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8081
      failureThreshold: 30    # 30 × 10s = 5 minutes for JVM to start
      periodSeconds: 10

    # Liveness probe — checked throughout lifetime; failure → restart
    livenessProbe:
      httpGet:
        path: /actuator/health/liveness
        port: 8081
      initialDelaySeconds: 0    # startup probe handles the delay
      periodSeconds: 10
      failureThreshold: 3

    # Readiness probe — failure → removed from load balancer
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: 8081
      initialDelaySeconds: 5
      periodSeconds: 5
      failureThreshold: 3
4

Metrics & Service Monitoring

The RED method (Rate, Errors, Duration) and USE method (Utilisation, Saturation, Errors) provide service- and infrastructure-level metrics for dashboards and alerts.

  • ✓RED method (Rate, Errors, Duration) measures service-level performance from user perspective — apply to every API endpoint
  • ✓USE method (Utilisation, Saturation, Errors) measures resource-level performance — apply to CPU, thread pools, connection pools
  • ✓Spring Boot Actuator + micrometer-registry-prometheus auto-instruments HTTP, JVM, HikariCP, and Kafka metrics at zero cost
  • ✓Enable percentiles-histogram in application.yml to allow PromQL histogram_quantile() for accurate P50/P95/P99 queries
  • ✓Tag all metrics with application name and environment — this enables dashboard filtering and multi-service comparison
  • ✓Alerts should have for: duration to avoid transient spike noise, and runbook links for on-call engineers
PromQL — RED and USE method queries for Spring Boot services
# PromQL examples for RED metrics (Spring Boot / Micrometer)

# Rate: requests per second (5-min average)
rate(http_server_requests_seconds_count{job="order-service"}[5m])

# Error rate: 5xx errors as % of total requests
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m])) * 100

# Duration: P99 latency for POST /orders
histogram_quantile(0.99,
  sum(rate(http_server_requests_seconds_bucket{
    uri="/orders", method="POST"}[5m])) by (le))

# USE metrics (infrastructure)
# Utilisation: JVM thread pool active vs max
hikaricp_connections_active / hikaricp_connections_max

# Saturation: pending (queued) HikariCP connection requests
hikaricp_connections_pending

# JVM memory utilisation
jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}
Learn this free with Aria, your AI tutor → AiCanCode.org/learn/microservices