Cloud storage and data management
Cloud storage and data management: concepts, architectures (object/block/file), durability vs availability, replication and erasure coding, consistency models, tiering and lifecycle, with formulas and worked exam-style examples.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Cloud engineers design and operate storage that holds user and application data reliably, cheaply and with required performance. Understanding trade-offs (durability, availability, cost, latency, consistency) is essential when choosing object, block or file storage, designing replication or erasure-coding schemes, and when answering exam questions about storage architectures and SLA calculations.
Key ideas
Start from first principles and define terms before using them.
Cloud storage — storage service provided over a network (the cloud) that stores and serves data remotely. It exposes APIs for applications and hides the physical hardware.
Object storage — a storage model where data is stored as discrete objects (data + metadata + unique ID). Good for large-scale, immutable or append-once content (backups, logs). Access is by API (e.g., REST GET/PUT).
Block storage — presents raw block devices (fixed-size blocks) to VMs or OS; the OS formats these blocks with a file system. Used for transactional databases and VM disks.
File storage — stores data in a hierarchical file system (directories and files) and is accessed over file protocols (NFS, SMB). Good for shared file access.
Durability — probability that data will not be lost (often measured over a year). Durable storage ensures data survives crashes and disk failures.
Availability — probability that the storage service responds to a client request successfully at a random time. Availability is about access; durability is about survival of data.
Replication — storing copies of the same data on multiple independent nodes to increase availability and durability.
Erasure coding — algorithmic method (like Reed-Solomon) that breaks data into k data fragments and m parity fragments; any k of (k+m) fragments can reconstruct the data. Reduces storage overhead compared to simple replication at the cost of CPU and network for reconstruction.
Consistency — guarantees about the order and visibility of reads/writes across replicas. Types include strong consistency (reads always return latest committed write) and eventual consistency (if no new writes, all replicas converge to same value eventually).
CAP theorem — in a distributed system that can suffer network partitions, you cannot simultaneously guarantee both strong consistency and availability during a partition; you must choose one while still tolerating the partition. Assumes an asynchronous network (unbounded message delay). In the absence of partitions, both consistency and availability are achievable. Also recall PACELC: if there is a Partition you trade Availability vs Consistency; Else you trade Latency vs Consistency. (See full statement and assumptions below.)
Tiering and lifecycle — classifying data as hot (frequently accessed, low latency), warm, cold or archive (rare access, high retrieval latency) and moving objects between tiers to reduce cost.
Metadata — data about data (object size, timestamps, custom tags). Object stores rely on metadata for search and lifecycle decisions.
Important exact statements and derivations universities ask for:
- Exact CAP theorem statement and assumptions
- Statement: In any distributed data system that must remain correct in the presence of network partitions, it is impossible during a partition to guarantee both (C) linearizability/strong consistency and (A) availability for every request to a non-failing node while also tolerating the partition. In the absence of partitions, both C and A can be provided.
- Assumptions: asynchronous network (unbounded message delay); availability means every request received by a non-failing node eventually results in a response. Note: independence of failures is not required by CAP.
- Availability of replicated system (derivation universities expect)
- Model: n independent replicas, replica i has availability A_i (probability replica i answers a request). System answers read if at least one replica is up (for read-any topology). So system availability A_system = 1 - probability(all replicas fail).
- For identical replicas with availability A: A_system = 1 - (1 - A)^n. (Derive by independence and complement rule.)
- Storage overhead and usable capacity
- Replication factor r: storage overhead = r× raw used. Usable fraction = 1/r.
- Erasure coding (k data shards, m parity shards): total shards = k + m; storage overhead = (k+m)/k; usable fraction = k/(k+m).
- Durability model for replication
- If each replica has probability p of surviving a time window independently, and we store r replicas, probability at least one copy survives = 1 - (1 - p)^r.
- Quorum replication (N, R, W)
- For strong consistency in quorum systems: R + W > N with R > 0 and W > 0. A read succeeds if at least R replicas respond; a write succeeds if at least W replicas respond. Under identical independent replica availability A, read availability A_read = sum_{k=R..N} C(N, k) A^k (1 - A)^{N - k}, and write availability A_write = sum_{k=W..N} C(N, k) A^k (1 - A)^{N - k}.
Diagrams to draw (describe exactly):
- Replication diagram: draw three boxes labelled Replica A, Replica B, Replica C; connect client to load balancer to all replicas; label network arrows; show master/leader arrow if using leader-based replication.
- Erasure coding layout: draw a rectangle split into k data blocks and m parity blocks arranged across different nodes; label reconstruction path: any k blocks reconstruct original.
- Consistency timeline: draw a time axis, show write W at t0 to replica 1, show read R at t1 to replica 2 returning old value under eventual consistency; label when convergence happens.
Cross-references: For virtualization, object vs block distinctions connect to "Storage and database as a service" and to "Cloud data centres and storage: object, block and file".
Formulas
A_system = 1 - ∏_{i=1..n} (1 - A_i) — A_system is system availability (dimensionless, fraction between 0 and 1); A_i is availability of replica i; applies when system succeeds if any replica responds; assumes independent replica failures.
A_system = 1 - (1 - A)^n — A is availability of each identical replica; n is number of replicas; dimensionless; derived from previous formula for identical replicas.
Usable_capacity_replication = Raw_capacity / r — Raw_capacity in bytes (e.g., PB); r is replication factor (dimensionless); applies for full replication.
Overhead_erasure = (k + m) / k — k data shards, m parity shards; overhead is dimensionless multiplier; usable fraction = k / (k + m).
Durability_replication = 1 - (1 - p)^r — p is per-replica survival probability over time window; r replicas; dimensionless; assumes independent failures.
Storage_cost = Usable_capacity × Cost_per_byte × Overhead — Usable_capacity in bytes to be stored; Cost_per_byte billed per raw byte; Overhead dimensionless multiplier.
Latency_total ≈ Latency_client_network + Latency_storage_node + (reconstruction_overhead) — times in seconds; approximation used in system-level latency estimates.
Quorum_condition: R + W > N — for strong consistency in quorum replication; integers with 0 < R, W ≤ N.
A_read_quorum = Σ_{k=R..N} C(N, k) A^k (1 - A)^{N - k} — identical independent replicas with per-replica availability A.
A_write_quorum = Σ_{k=W..N} C(N, k) A^k (1 - A)^{N - k} — identical independent replicas with per-replica availability A.
Worked examples
Example 1 — Availability with 3 identical replicas (numerical)
Given: three replicas, each with availability 99.9% = 0.999. Assume independence.
Formula: A_system = 1 - (1 - A)^n
Compute: (1 - A) = 0.001. (1 - A)^3 = 0.001^3 = 1e-9.
So A_system = 1 - 1e-9 = 0.999999999 = 99.9999999%.
Answer: System availability ≈ 99.9999999% (nine nines).
Example 2 — Usable capacity: replication vs erasure coding (numerical) Given: Raw storage 1.0 PB (1 PB = 10^15 bytes for approximate accounting; use your university's convention), compare 3× replication vs erasure coding with k=6, m=3 (total shards 9). Replication usable capacity = Raw / 3 = 1.0 PB / 3 = 0.333... PB. Erasure coding usable fraction = k / (k+m) = 6 / 9 = 2/3. Usable capacity = Raw × 2/3 = 0.666... PB. Answer: With same raw 1 PB, erasure coding (6+3) gives ~0.667 PB usable, replication (3×) gives ~0.333 PB usable.
Example 3 — Consistency scenario (theory model answer) Question: Explain with a timeline how eventual consistency can return stale reads and how strong consistency avoids that. Model answer: Draw time axis. At t0 client A writes value X to leader; leader acknowledges after replicating to its local disk but before replica B has applied it. At t1 client B reads from replica B under eventual consistency; replica B has old value Y and returns Y (stale read). Under strong consistency (linearizability), reads would be routed or synchronized so that read at t1 must see X, e.g., by contacting leader or ensuring quorum reads that intersect the write quorum. State the cost: strong consistency increases read latency or reduces availability in partitions. Answer: Eventual consistency allows stale reads; strong consistency prevents them at higher latency or availability cost.
Common mistakes
- Thinking erasure coding always reduces latency: it reduces storage overhead but increases CPU/network cost for writes and reconstructs, so latency can increase for writes and degraded reads.
- Misapplying CAP theorem: CAP says you cannot have all three in presence of partitions; it does not say one must always choose only C or A—design trade-offs depend on expected partitions.
- Mixing percentages: be careful using "nines"; a small change in exponent leads to large changes in perceived reliability.
For your exam
Types of questions:
- Definitions: define object/block/file storage, durability, availability, consistency, replication, erasure coding.
- Short notes: compare object vs block storage; explain tiering; describe CAP theorem with implications.
- Derivations: derive availability of n replicas; compute usable capacity under replication and erasure coding.
- Numericals: given replica availability or raw capacity, compute system availability or usable capacity.
- Diagrams: draw replication and erasure coding layouts and label components.
- Long-answer/theory: explain consistency models, show timeline and state trade-offs between consistency, availability, latency and cost.
What full-marks answers include:
- Clear definitions (bold the term in your answer if the university allows). Include assumptions (e.g., independence of failures).
- Correct formula and step-by-step derivation, carrying units and showing final numeric value with correct precision. Label diagrams clearly (nodes, network, leader, shards).
- Mention practical trade-offs: cost vs performance, reconstruction cost for erasure coding, operational complexity (metadata servers, object indexes), and where each storage model is used (object: backups, CDN; block: VM disks; file: shared files).
Quick check
- Define durability in one sentence.
- For 4 identical replicas each 99% available, what is system availability (read-any)?
- For erasure coding with k=4, m=2, what fraction of raw capacity is usable?
Answers: durability is probability data survives over a time window (no data loss); availability = 1 - (1 - 0.99)^4 = 1 - 0.01^4 = 1 - 1e-8 = 0.99999999; usable fraction = k/(k+m) = 4/6 = 2/3.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?