Home/Learn/System Design/Write-Ahead Log (WAL)

Write-Ahead Log (WAL)

Advanced
Storage & File Systems

A write-ahead log records every change to an append-only log on disk before applying it to the main data structure. It guarantees durability and crash recovery in databases, message brokers, and distributed systems.

Overview

A Write-Ahead Log (WAL) is a fundamental technique for ensuring durability. Before any change is made to the actual data structures (B-Tree pages, in-memory state), the change is first written to a sequential, append-only log file on disk. If the system crashes after writing to the WAL but before updating the main data, it can replay the WAL on restart to recover the changes. WAL exploits the fact that sequential writes are orders of magnitude faster than random writes on both HDDs and SSDs. Every major database uses WAL: PostgreSQL (WAL), MySQL (redo log), SQLite (WAL mode), and MongoDB (journal). Beyond databases, WAL is used in Kafka (append-only commit log), Raft/Paxos consensus (replicated log), and event sourcing (event store). WAL is the backbone of durability, replication, and point-in-time recovery.

How WAL Works

Every write is first appended to the WAL (sequential I/O — fast). Then the actual data pages are updated in memory. Periodically, dirty pages are flushed to disk (checkpoint). On crash, replay WAL from last checkpoint.

Conceptual — WAL operation and crash recovery
// WAL operation sequence
//
// 1. Client: INSERT INTO orders VALUES (...)
// 2. DB appends to WAL: "INSERT order-123, {data...}" → fsync to disk ✅
// 3. DB updates in-memory page (B-Tree leaf)
// 4. Client receives "commit OK" (durable — in WAL)
// 5. Background: dirty pages flushed to disk (checkpoint)
//
// Crash after step 2, before step 5:
// → Restart: replay WAL from last checkpoint → data recovered ✅
//
// Crash before step 2:
// → Transaction lost (not committed) → correct behaviour ✅

// PostgreSQL WAL
// Location: pg_wal/ directory
// Segment files: 16 MB each (default)
// WAL level:
//   minimal    — crash recovery only
//   replica    — crash recovery + replication
//   logical    — + logical decoding (CDC)

// WAL performance: sequential writes are 100x faster than random
// HDD: sequential write ~200 MB/s vs random write ~2 MB/s
// SSD: sequential write ~3 GB/s vs random write ~500 MB/s

WAL Beyond Databases

WAL is used in Kafka (commit log), consensus protocols (Raft replicated log), and event sourcing. The pattern is universal: append to a durable log before taking action.

Conceptual — WAL in Kafka, Raft, and event sourcing
// Kafka: the entire system IS a WAL
// Topics are append-only logs split into partitions
// Producers append records; consumers read at their own pace
// Records retained for configurable period (not deleted on read)
// Replication: WAL is replicated to follower brokers

// Raft consensus: replicated WAL
// Leader appends command to local log
// Leader replicates log entry to followers
// Once majority ACK → entry is committed
// All nodes apply committed entries to state machine
//
// Log: [1: set x=1] [2: set y=2] [3: del x] [4: set z=3]
//       committed     committed   committed   uncommitted

// Event sourcing: event store is a WAL
// Every state change is an append-only event
// Current state = replay(events)
// Snapshots = checkpoints (avoid replaying entire history)

Key Points to Remember

  • 1WAL writes every change to a sequential, append-only log before updating main data structures.
  • 2Sequential writes are 100x+ faster than random writes — WAL exploits this for performance.
  • 3On crash, replay WAL from last checkpoint to recover committed but unflushed changes.
  • 4Used by all major databases (PostgreSQL WAL, MySQL redo log, MongoDB journal).
  • 5Same pattern in Kafka (commit log), Raft (replicated log), and event sourcing (event store).

Interview Questions

Sign in to ask Aria
1

What is a write-ahead log and why is it used?

EasyTCS
2

How does WAL provide crash recovery in PostgreSQL?

MediumAmazon
3

Why are sequential writes faster than random writes?

MediumGoogle
4

How does Kafka use the write-ahead log concept?

MediumFlipkart
5

Design a durable message queue using write-ahead logging.

HardUber

Ask Aria about Write-Ahead Log (WAL)

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Sign in to save personal notes for this topic.

Discussion

Sign in to join the discussion.

Loading discussion…