Latency vs Throughput
BeginnerLatency is the time to complete a single request; throughput is the number of requests processed per unit time. Optimising one often impacts the other — understanding this trade-off is fundamental to system design.
Overview
Latency measures how long a single operation takes (e.g. 50ms for an API call). Throughput measures how many operations the system handles per second (e.g. 10,000 requests/second). They are related but not inversely proportional — you can sometimes improve both. For example, batching increases throughput but increases individual request latency. Connection pooling improves both by reducing connection setup overhead. In system design interviews, you need to identify which metric matters more for the use case: a real-time chat system prioritises low latency; a batch ETL pipeline prioritises high throughput. Key latency numbers every engineer should know: L1 cache (~1ns), RAM (~100ns), SSD read (~100μs), network round-trip same DC (~500μs), cross-region (~50-150ms).
Key Latency Numbers
Knowing order-of-magnitude latency for common operations helps you estimate system performance and identify bottlenecks during design.
// Latency numbers every programmer should know (2024)
//
// Operation | Latency
// ─────────────────────────────────────────
// L1 cache reference | 1 ns
// L2 cache reference | 4 ns
// RAM reference | 100 ns
// SSD random read | 100 μs
// HDD seek | 4,000 μs (4 ms)
// Network: same datacenter | 500 μs (0.5 ms)
// Network: cross-region | ~80 ms
// Network: cross-continent | ~150 ms
// Redis GET | ~0.5 ms
// PostgreSQL simple query | ~2 ms
// S3 GET (same region) | ~20 ms
// External API call | 50-500 ms
// Rule of thumb: memory is 1000x faster than disk,
// same-DC network is 100x faster than cross-region.Trade-offs & Optimisation
Batching improves throughput at the cost of latency. Caching improves both. Parallelism improves throughput. Connection pooling reduces per-request overhead. The right optimisation depends on your SLA.
// Throughput vs Latency trade-offs
//
// Technique | Throughput | Latency | Notes
// ───────────────────────────────────────────────────
// Batching | ↑↑ | ↑ (worse)| Kafka producer linger.ms
// Caching | ↑ | ↓ (better)| Best of both worlds
// Compression | ↑ (net) | ↑ (CPU) | Saves bandwidth, costs CPU
// Connection pool | ↑ | ↓ | Avoid TCP/TLS setup per req
// Parallelism | ↑↑ | ↔ | More threads/goroutines
// Async I/O | ↑↑ | ↔ | Non-blocking event loop
// Example: Kafka batching trade-off
# High throughput (batch): linger 50ms, batch 64KB
linger.ms=50
batch.size=65536
# Result: ~200K msgs/sec, p99 latency ~80ms
# Low latency: linger 0ms
linger.ms=0
batch.size=16384
# Result: ~50K msgs/sec, p99 latency ~5msKey Points to Remember
- 1Latency = time for one operation; throughput = operations per unit time.
- 2Know key latency numbers: RAM ~100ns, SSD ~100μs, same-DC network ~500μs, cross-region ~80ms.
- 3Batching improves throughput but increases latency; caching can improve both.
- 4SLAs should define both latency (p50, p95, p99) and throughput (RPS) targets.
- 5Tail latency (p99, p99.9) matters more than average in distributed systems.
Interview Questions
Sign in to ask AriaWhat is the difference between latency and throughput?
What are some common latency numbers a system designer should know?
How does batching improve throughput at the cost of latency?
Why is p99 latency more important than average latency?
Design a system that must serve 100K RPS with p99 < 50ms.
Ask Aria about Latency vs Throughput
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.