Home/Learn/System Design/Logging Best Practices

Logging Best Practices

Beginner

Observability & Operations

Structured, centralised logging with correlation IDs is essential for debugging distributed systems. Log at appropriate levels, include context, and aggregate logs in a searchable platform (ELK, Loki).

Overview

In a monolith, you tail one log file. In microservices, a single request flows through multiple services, each with its own logs. Effective logging requires: (1) Structured logs — JSON format with consistent fields (timestamp, service, level, message, traceId) instead of unstructured text. (2) Correlation IDs — a unique ID propagated across all services in a request chain, enabling you to find all logs for a single user action. (3) Centralised aggregation — ship logs from all services to a central platform (ELK stack: Elasticsearch + Logstash + Kibana, or Grafana Loki). (4) Appropriate log levels — ERROR for failures, WARN for degradation, INFO for business events, DEBUG for troubleshooting (off in production). (5) Context — include userId, requestId, orderId in every log line. Avoid logging sensitive data (passwords, tokens, PII).

Structured Logging

JSON-formatted logs are machine-parseable and searchable. Include standard fields: timestamp, level, service name, trace ID, and contextual data.

JSON + XML + Java — structured logging

// Unstructured log (bad — hard to parse and search)
2025-03-29 10:00:01 INFO OrderService - Order 123 placed by user 42

// Structured log (good — JSON, searchable)
{
  "timestamp": "2025-03-29T10:00:01.234Z",
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc-123-def-456",
  "spanId": "span-789",
  "userId": "u-42",
  "orderId": "order-123",
  "message": "Order placed successfully",
  "total": 99.99,
  "itemCount": 3,
  "durationMs": 45
}

// Spring Boot + Logback JSON encoder
// logback-spring.xml
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>traceId</includeMdcKeyName>
    <includeMdcKeyName>userId</includeMdcKeyName>
  </encoder>
</appender>

// Using MDC for context propagation
MDC.put("userId", currentUser.getId());
MDC.put("orderId", order.getId());
log.info("Order placed successfully");

Centralised Log Aggregation

Ship logs from all services to a central platform. ELK (Elasticsearch + Logstash + Kibana) and Grafana Loki are the most common stacks.

Conceptual — log aggregation and best practices

// Centralised logging architecture
//
// Service A ──┐
// Service B ──┤──► Log Shipper ──► Log Aggregator ──► Search/Dashboard
// Service C ──┘   (Filebeat,       (Elasticsearch,    (Kibana,
//                  Fluentd)         Loki)               Grafana)
//
// ELK Stack:
// Filebeat (agent) → Logstash (parse/transform) → Elasticsearch (store/index)
//                                                → Kibana (search/visualise)
//
// Grafana Loki (lighter weight):
// Promtail (agent) → Loki (store, label-indexed) → Grafana (search/visualise)

// Log levels — use appropriately
// ERROR: something broke, needs attention (page on-call)
// WARN:  degraded state, fallback used (monitor)
// INFO:  business events (order placed, user signed up)
// DEBUG: detailed troubleshooting (off in production!)
// TRACE: extreme detail (never in production)

// What NOT to log:
// ❌ Passwords, API keys, tokens
// ❌ Full credit card numbers (PCI compliance)
// ❌ PII without masking (GDPR)
// ❌ Large payloads (base64 encoded files)

Key Points to Remember

1Structured JSON logs with consistent fields are searchable and machine-parseable.
2Correlation IDs (traceId) propagated across services link all logs for a single request.
3Centralise logs in ELK or Grafana Loki — never rely on SSH-ing into individual servers.
4Use appropriate log levels: ERROR for failures, INFO for business events, DEBUG off in production.
5Never log passwords, tokens, credit card numbers, or unmasked PII.

Interview Questions

Why is structured logging important in microservices?

EasyInfosys

What is a correlation ID and how is it propagated across services?

MediumAmazon

Compare ELK stack and Grafana Loki for log aggregation.

MediumGoogle

How do you handle logging at scale (1 TB of logs per day)?

HardFlipkart

Design a centralised logging platform for 200 microservices.

HardNetflix

Ask Aria about Logging Best Practices

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Design: Rate Limiter

Monitoring & Alerting