Distributed File Systems
AdvancedBy the AiCanCode.org team · How we write and check our content
Distributed file systems (HDFS, GFS, Ceph) store data across many machines, providing fault tolerance through replication and enabling parallel processing of massive datasets.
Overview
A distributed file system (DFS) spreads data across multiple servers while presenting a unified namespace to clients. Google File System (GFS) and its open-source clone HDFS (Hadoop Distributed File System) pioneered this for big data processing. HDFS splits files into large blocks (128 MB default), replicates each block across multiple DataNodes (default 3 replicas), and uses a NameNode to manage metadata. This design enables: fault tolerance (block replicas survive node failures), parallel processing (MapReduce/Spark read blocks from local DataNodes), horizontal scaling (add DataNodes for more storage), and cost-effectiveness (commodity hardware). Modern alternatives include Ceph (POSIX-compatible, unified storage), MinIO (S3-compatible, cloud-native), and cloud-native options (S3, GCS) that have largely replaced HDFS for new workloads.
HDFS Architecture
HDFS splits files into blocks distributed across DataNodes. A NameNode manages the metadata (which blocks belong to which file, where blocks are stored). Blocks are replicated for fault tolerance.
// HDFS architecture
//
// Client
// │ 1. "Write file.csv" → NameNode (metadata)
// │ 2. NameNode returns: block1→[DN1,DN3,DN5], block2→[DN2,DN4,DN6]
// │ 3. Client writes block1 to DN1 → DN1 replicates to DN3, DN5
// │ Client writes block2 to DN2 → DN2 replicates to DN4, DN6
//
// NameNode (master)
// ├── File: /data/file.csv
// │ ├── Block1 (128MB): DN1, DN3, DN5
// │ └── Block2 (128MB): DN2, DN4, DN6
//
// DataNode1 DataNode2 DataNode3 DataNode4 DataNode5 DataNode6
// [block1] [block2] [block1] [block2] [block1] [block2]
//
// If DN1 fails: block1 still available on DN3 and DN5
// NameNode detects failure → replicates block1 to DN4 (maintain 3 replicas)
// HDFS config
dfs.replication = 3 // 3 copies of every block
dfs.blocksize = 134217728 // 128 MB block size
dfs.namenode.name.dir = ... // NameNode metadata locationModern Alternatives
Ceph provides unified storage (block, file, object). MinIO offers S3-compatible object storage for on-premises. Cloud-native storage (S3, GCS) has replaced HDFS for many workloads.
// Ceph — unified distributed storage
// Stores data as objects in RADOS (Reliable Autonomic Distributed Object Store)
// Provides three interfaces:
// CephFS — POSIX file system (replaces HDFS)
// RBD — block storage (replaces SAN)
// RGW — S3-compatible object storage
//
// Uses CRUSH algorithm (not consistent hashing) for data placement
// Self-healing: automatically rebalances when nodes join/leave
// MinIO — S3-compatible, Kubernetes-native
// Deploy on-premises with S3 API compatibility
// Erasure coding (instead of replication) for space efficiency
// Used for: data lakes, backup, ML model storage
// When to use what:
// HDFS: Legacy Hadoop ecosystem, MapReduce/Spark jobs
// Ceph: On-premises, need block + file + object storage
// MinIO: On-premises, S3 compatibility required
// S3/GCS: Cloud-native, virtually unlimited scale, managedKey Points to Remember
- 1Distributed file systems split files into blocks replicated across multiple machines.
- 2HDFS: NameNode (metadata) + DataNodes (data blocks) — designed for batch processing with large files.
- 3Block replication (default 3x) provides fault tolerance — data survives node failures.
- 4Ceph provides unified distributed storage: file, block, and object interfaces.
- 5Cloud object storage (S3, GCS) has replaced HDFS for most new workloads.
Interview Questions
Sign in to ask AriaHow does HDFS store and replicate data?
What happens when a DataNode fails in HDFS?
Why is the NameNode a single point of failure and how is it solved?
Compare HDFS, Ceph, and S3 for a data lake use case.
Design a distributed file system for storing 10 PB of video data.
Ask Aria about Distributed File Systems
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to ask a question — Aria answers, and so do other students.