Storage & File Systems — Cheat Sheet

Blob & Object Storage

// S3 object structure
// Bucket: my-app-assets
// Key: uploads/users/u-42/avatar.jpg
// → Not a real folder structure, just a key string
//
// Object has:
//   Key:      uploads/users/u-42/avatar.jpg
//   Data:     (binary content)
//   Metadata: Content-Type: image/jpeg, x-amz-meta-user-id: u-42
//   Size:     up to 5 TB per object
//   ETag:     MD5 hash for integrity

// S3 durability: 99.999999999% (11 nines)
// = losing 1 object out of 100 billion per year
// Achieved via automatic replication across 3+ AZs

// AWS SDK upload example (Java)
s3Client.putObject(
    PutObjectRequest.builder()
        .bucket("my-app-assets")
        .key("uploads/users/u-42/avatar.jpg")
        .contentType("image/jpeg")
        .build(),
    RequestBody.fromFile(avatarFile)
);

Distributed File Systems

// HDFS architecture
//
//  Client
//    │ 1. "Write file.csv" → NameNode (metadata)
//    │ 2. NameNode returns: block1→[DN1,DN3,DN5], block2→[DN2,DN4,DN6]
//    │ 3. Client writes block1 to DN1 → DN1 replicates to DN3, DN5
//    │    Client writes block2 to DN2 → DN2 replicates to DN4, DN6
//
//  NameNode (master)
//  ├── File: /data/file.csv
//  │   ├── Block1 (128MB): DN1, DN3, DN5
//  │   └── Block2 (128MB): DN2, DN4, DN6
//
//  DataNode1  DataNode2  DataNode3  DataNode4  DataNode5  DataNode6
//  [block1]   [block2]   [block1]   [block2]   [block1]   [block2]
//
// If DN1 fails: block1 still available on DN3 and DN5
// NameNode detects failure → replicates block1 to DN4 (maintain 3 replicas)

// HDFS config
dfs.replication = 3           // 3 copies of every block
dfs.blocksize = 134217728     // 128 MB block size
dfs.namenode.name.dir = ...   // NameNode metadata location

Data Lake vs Data Warehouse

// Data warehouse architecture
//
// Source Systems → ETL Pipeline → Data Warehouse → BI Tools
// (MySQL, APIs)   (Airflow,       (Snowflake,      (Tableau,
//                  dbt)            Redshift)        Looker)
//
// Star schema:
//   Fact table: sales (sale_id, date_id, product_id, amount)
//   Dimension: dim_date (date_id, year, quarter, month)
//   Dimension: dim_product (product_id, name, category)

// Snowflake query — fast analytics on structured data
SELECT d.year, d.quarter, p.category,
       SUM(s.amount) AS revenue, COUNT(*) AS transactions
FROM sales s
JOIN dim_date d ON s.date_id = d.date_id
JOIN dim_product p ON s.product_id = p.product_id
WHERE d.year = 2025
GROUP BY d.year, d.quarter, p.category
ORDER BY revenue DESC;

// Characteristics:
// ✅ Fast queries (columnar storage, pre-optimised)
// ✅ Schema-on-write (data is clean and structured)
// ❌ Expensive storage
// ❌ Only handles structured data

Write-Ahead Log (WAL)

// WAL operation sequence
//
// 1. Client: INSERT INTO orders VALUES (...)
// 2. DB appends to WAL: "INSERT order-123, {data...}" → fsync to disk ✅
// 3. DB updates in-memory page (B-Tree leaf)
// 4. Client receives "commit OK" (durable — in WAL)
// 5. Background: dirty pages flushed to disk (checkpoint)
//
// Crash after step 2, before step 5:
// → Restart: replay WAL from last checkpoint → data recovered ✅
//
// Crash before step 2:
// → Transaction lost (not committed) → correct behaviour ✅

// PostgreSQL WAL
// Location: pg_wal/ directory
// Segment files: 16 MB each (default)
// WAL level:
//   minimal    — crash recovery only
//   replica    — crash recovery + replication
//   logical    — + logical decoding (CDC)

// WAL performance: sequential writes are 100x faster than random
// HDD: sequential write ~200 MB/s vs random write ~2 MB/s
// SSD: sequential write ~3 GB/s vs random write ~500 MB/s

LSM-Tree vs B-Tree

// B-Tree structure (simplified)
//
//              [50]                    ← root page
//             /    \
//        [20,30]   [70,80]            ← internal pages
//        / | \     / | \
//      [...] [...] [...] [...]         ← leaf pages (data)
//
// Read: traverse root → internal → leaf = O(log n)
//   4 levels can index ~1 billion rows (branching factor ~500)
//
// Write: find leaf page → update in place → write page to disk
//   Random I/O (write to specific page location)
//   WAL write first for durability
//
// Strengths:
// ✅ Fast reads — O(log n), few disk seeks
// ✅ Efficient range queries — leaves are linked
// ✅ Predictable performance
//
// Weaknesses:
// ❌ Write amplification — updating one row writes entire page
// ❌ Random I/O for writes
// ❌ Page splits when full (rebalancing)