Distributed File Systems
AdvancedDistributed file systems (HDFS, GFS, Ceph) store data across many machines, providing fault tolerance through replication and enabling parallel processing of massive datasets.
Overview
A distributed file system (DFS) spreads data across multiple servers while presenting a unified namespace to clients. Google File System (GFS) and its open-source clone HDFS (Hadoop Distributed File System) pioneered this for big data processing. HDFS splits files into large blocks (128 MB default), replicates each block across multiple DataNodes (default 3 replicas), and uses a NameNode to manage metadata. This design enables: fault tolerance (block replicas survive node failures), parallel processing (MapReduce/Spark read blocks from local DataNodes), horizontal scaling (add DataNodes for more storage), and cost-effectiveness (commodity hardware). Modern alternatives include Ceph (POSIX-compatible, unified storage), MinIO (S3-compatible, cloud-native), and cloud-native options (S3, GCS) that have largely replaced HDFS for new workloads.
HDFS Architecture
HDFS splits files into blocks distributed across DataNodes. A NameNode manages the metadata (which blocks belong to which file, where blocks are stored). Blocks are replicated for fault tolerance.
// HDFS architecture
//
// Client
// │ 1. "Write file.csv" → NameNode (metadata)
// │ 2. NameNode returns: block1→[DN1,DN3,DN5], block2→[DN2,DN4,DN6]
// │ 3. Client writes block1 to DN1 → DN1 replicates to DN3, DN5
// │ Client writes block2 to DN2 → DN2 replicates to DN4, DN6
//
// NameNode (master)
// ├── File: /data/file.csv
// │ ├── Block1 (128MB): DN1, DN3, DN5
// │ └── Block2 (128MB): DN2, DN4, DN6
//
// DataNode1 DataNode2 DataNode3 DataNode4 DataNode5 DataNode6
// [block1] [block2] [block1] [block2] [block1] [block2]
//
// If DN1 fails: block1 still available on DN3 and DN5
// NameNode detects failure → replicates block1 to DN4 (maintain 3 replicas)
// HDFS config
dfs.replication = 3 // 3 copies of every block
dfs.blocksize = 134217728 // 128 MB block size
dfs.namenode.name.dir = ... // NameNode metadata locationModern Alternatives
Ceph provides unified storage (block, file, object). MinIO offers S3-compatible object storage for on-premises. Cloud-native storage (S3, GCS) has replaced HDFS for many workloads.
// Ceph — unified distributed storage
// Stores data as objects in RADOS (Reliable Autonomic Distributed Object Store)
// Provides three interfaces:
// CephFS — POSIX file system (replaces HDFS)
// RBD — block storage (replaces SAN)
// RGW — S3-compatible object storage
//
// Uses CRUSH algorithm (not consistent hashing) for data placement
// Self-healing: automatically rebalances when nodes join/leave
// MinIO — S3-compatible, Kubernetes-native
// Deploy on-premises with S3 API compatibility
// Erasure coding (instead of replication) for space efficiency
// Used for: data lakes, backup, ML model storage
// When to use what:
// HDFS: Legacy Hadoop ecosystem, MapReduce/Spark jobs
// Ceph: On-premises, need block + file + object storage
// MinIO: On-premises, S3 compatibility required
// S3/GCS: Cloud-native, virtually unlimited scale, managedKey Points to Remember
- 1Distributed file systems split files into blocks replicated across multiple machines.
- 2HDFS: NameNode (metadata) + DataNodes (data blocks) — designed for batch processing with large files.
- 3Block replication (default 3x) provides fault tolerance — data survives node failures.
- 4Ceph provides unified distributed storage: file, block, and object interfaces.
- 5Cloud object storage (S3, GCS) has replaced HDFS for most new workloads.
Interview Questions
Sign in to ask AriaHow does HDFS store and replicate data?
What happens when a DataNode fails in HDFS?
Why is the NameNode a single point of failure and how is it solved?
Compare HDFS, Ceph, and S3 for a data lake use case.
Design a distributed file system for storing 10 PB of video data.
Ask Aria about Distributed File Systems
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.