Home/Learn/Microservices/Centralised Logging (ELK/Loki)

Centralised Logging (ELK/Loki)

Intermediate

Observability

All services ship structured JSON logs to an aggregator (Elasticsearch, Loki); correlation IDs link logs across service boundaries for end-to-end debugging.

Overview

In a microservices architecture, a single user request touches dozens of services. Without centralised logging, debugging requires SSH-ing into individual pods and grepping logs in isolation — impossible at scale. The solution is structured logging: every service emits JSON to stdout; a sidecar or DaemonSet (Fluent Bit, Promtail) forwards logs to a centralised store (Elasticsearch or Grafana Loki). The critical enabler is a correlation ID (also called trace ID): a UUID generated at the API gateway and propagated via HTTP headers (X-Correlation-Id) and Kafka message headers through every downstream call. Logback's MDC (Mapped Diagnostic Context) attaches the ID to every log line automatically, making it trivial to filter all logs for a single request across 10 services in Kibana or Grafana Loki.

Structured JSON logging with Logback + MDC

Replace the default Logback pattern with logstash-logback-encoder to emit JSON. A Spring filter or interceptor seeds MDC with the incoming correlation ID so all log lines within the request automatically include it.

Java — Logback JSON + MDC correlation filter

<!-- pom.xml -->
<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>7.4</version>
</dependency>

<!-- logback-spring.xml -->
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>correlationId</includeMdcKeyName>
    <includeMdcKeyName>userId</includeMdcKeyName>
  </encoder>
</appender>

// CorrelationFilter.java
@Component
@Order(1)
public class CorrelationFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req,
            HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String id = Optional.ofNullable(req.getHeader("X-Correlation-Id"))
                            .orElse(UUID.randomUUID().toString());
        MDC.put("correlationId", id);
        res.setHeader("X-Correlation-Id", id);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}

Propagating correlation IDs across Kafka

HTTP headers don't cross Kafka boundaries automatically. Attach the correlation ID as a Kafka message header on the producer side and extract it into MDC on the consumer side.

Java — Kafka header propagation

// Producer: attach header
kafkaTemplate.send(new ProducerRecord<>("orders", null, key, value,
    List.of(new RecordHeader("X-Correlation-Id",
        MDC.get("correlationId").getBytes(StandardCharsets.UTF_8)))));

// Consumer: extract header into MDC
@KafkaListener(topics = "orders")
public void handle(ConsumerRecord<String, OrderEvent> record) {
    Header header = record.headers().lastHeader("X-Correlation-Id");
    if (header != null) {
        MDC.put("correlationId", new String(header.value()));
    }
    try {
        processOrder(record.value());
    } finally {
        MDC.clear();
    }
}

Querying logs in Grafana Loki

Promtail or Fluent Bit ships logs to Loki; labels (service, namespace, pod) are indexed. Use LogQL to filter by correlation ID across all services.

YAML + LogQL — Loki config and query

# Promtail config snippet — scrape k8s pod logs
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - json:
          expressions:
            correlationId: correlationId
            level: level
      - labels:
          correlationId:
          level:

# LogQL query in Grafana
{namespace="production"} | json | correlationId="abc-123-xyz"

# Cross-service timeline for one request
{namespace="production", service=~"order-service|payment-service|inventory-service"}
  | json | correlationId="abc-123-xyz"
  | line_format "{{.service}} {{.level}} {{.message}}"

Key Points to Remember

1Always emit structured JSON — free-text logs cannot be reliably parsed or indexed.
2Correlation IDs must be generated at the edge (API gateway) and propagated through every HTTP call and message header.
3Use MDC.put() + MDC.clear() in a filter so all log statements in a request automatically include the ID without explicit passing.
4Logstash/Loki labels should be low-cardinality (service, env, level); correlation ID belongs in the log body, not as a label.
5Log level discipline: ERROR for actionable failures, WARN for degraded state, INFO for business events, DEBUG for dev only.
6Use log sampling at high QPS to control ingestion costs — only sample DEBUG and INFO, never sample ERROR.

Interview Questions

How does MDC work in a multithreaded Spring application and what are its pitfalls?

MediumNetflix

How would you propagate a correlation ID through an async @Async method or CompletableFuture?

HardUber

What is the difference between logs, metrics, and traces in observability (the three pillars)?

EasyDatadog

Why should you avoid high-cardinality values like user IDs as Loki/Prometheus labels?

MediumGrafana Labs

How would you implement log-based alerting — trigger a PagerDuty alert when ERROR rate spikes?

MediumAmazon

Ask Aria about Centralised Logging (ELK/Loki)

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Distributed Tracing

Health Checks & Readiness Probes