Home/Learn/System Design/Distributed File Systems

Distributed File Systems

Advanced
Storage & File Systems

By the AiCanCode.org team · How we write and check our content

Distributed file systems (HDFS, GFS, Ceph) store data across many machines, providing fault tolerance through replication and enabling parallel processing of massive datasets.

Overview

A distributed file system (DFS) spreads data across multiple servers while presenting a unified namespace to clients. Google File System (GFS) and its open-source clone HDFS (Hadoop Distributed File System) pioneered this for big data processing. HDFS splits files into large blocks (128 MB default), replicates each block across multiple DataNodes (default 3 replicas), and uses a NameNode to manage metadata. This design enables: fault tolerance (block replicas survive node failures), parallel processing (MapReduce/Spark read blocks from local DataNodes), horizontal scaling (add DataNodes for more storage), and cost-effectiveness (commodity hardware). Modern alternatives include Ceph (POSIX-compatible, unified storage), MinIO (S3-compatible, cloud-native), and cloud-native options (S3, GCS) that have largely replaced HDFS for new workloads.

HDFS Architecture

HDFS splits files into blocks distributed across DataNodes. A NameNode manages the metadata (which blocks belong to which file, where blocks are stored). Blocks are replicated for fault tolerance.

Conceptual + Config — HDFS architecture
// HDFS architecture
//
//  Client
//    │ 1. "Write file.csv" → NameNode (metadata)
//    │ 2. NameNode returns: block1→[DN1,DN3,DN5], block2→[DN2,DN4,DN6]
//    │ 3. Client writes block1 to DN1 → DN1 replicates to DN3, DN5
//    │    Client writes block2 to DN2 → DN2 replicates to DN4, DN6
//
//  NameNode (master)
//  ├── File: /data/file.csv
//  │   ├── Block1 (128MB): DN1, DN3, DN5
//  │   └── Block2 (128MB): DN2, DN4, DN6
//
//  DataNode1  DataNode2  DataNode3  DataNode4  DataNode5  DataNode6
//  [block1]   [block2]   [block1]   [block2]   [block1]   [block2]
//
// If DN1 fails: block1 still available on DN3 and DN5
// NameNode detects failure → replicates block1 to DN4 (maintain 3 replicas)

// HDFS config
dfs.replication = 3           // 3 copies of every block
dfs.blocksize = 134217728     // 128 MB block size
dfs.namenode.name.dir = ...   // NameNode metadata location

Modern Alternatives

Ceph provides unified storage (block, file, object). MinIO offers S3-compatible object storage for on-premises. Cloud-native storage (S3, GCS) has replaced HDFS for many workloads.

Conceptual — modern distributed storage options
// Ceph — unified distributed storage
// Stores data as objects in RADOS (Reliable Autonomic Distributed Object Store)
// Provides three interfaces:
//   CephFS  — POSIX file system (replaces HDFS)
//   RBD     — block storage (replaces SAN)
//   RGW     — S3-compatible object storage
//
// Uses CRUSH algorithm (not consistent hashing) for data placement
// Self-healing: automatically rebalances when nodes join/leave

// MinIO — S3-compatible, Kubernetes-native
// Deploy on-premises with S3 API compatibility
// Erasure coding (instead of replication) for space efficiency
// Used for: data lakes, backup, ML model storage

// When to use what:
// HDFS:       Legacy Hadoop ecosystem, MapReduce/Spark jobs
// Ceph:       On-premises, need block + file + object storage
// MinIO:      On-premises, S3 compatibility required
// S3/GCS:     Cloud-native, virtually unlimited scale, managed

Key Points to Remember

  • 1Distributed file systems split files into blocks replicated across multiple machines.
  • 2HDFS: NameNode (metadata) + DataNodes (data blocks) — designed for batch processing with large files.
  • 3Block replication (default 3x) provides fault tolerance — data survives node failures.
  • 4Ceph provides unified distributed storage: file, block, and object interfaces.
  • 5Cloud object storage (S3, GCS) has replaced HDFS for most new workloads.

Interview Questions

Sign in to ask Aria
1

How does HDFS store and replicate data?

EasyTCS
2

What happens when a DataNode fails in HDFS?

MediumAmazon
3

Why is the NameNode a single point of failure and how is it solved?

MediumGoogle
4

Compare HDFS, Ceph, and S3 for a data lake use case.

HardFlipkart
5

Design a distributed file system for storing 10 PB of video data.

HardNetflix

Ask Aria about Distributed File Systems

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Sign in to save personal notes for this topic.

Discussion

Sign in to ask a question — Aria answers, and so do other students.

Loading discussion…