How Observability Works
IntermediateObservability is the ability to understand what is happening inside a system from the data it emits — without shipping new code to investigate. It rests on three pillars: metrics (numeric measurements over time), logs (timestamped records of events), and traces (the path of a request across services). Traditional monitoring watches for known problems; observability lets you ask new questions and debug the "unknown unknowns" that plague complex distributed systems.
Think of a hospital patient monitor plus a full chart
Monitoring is the bedside machine beeping when heart rate crosses a preset alarm — it catches the problems you already anticipated. Observability is having heart rate, oxygen, bloodwork, and a full timeline of everything that happened, so when the patient shows a strange new symptom, doctors can investigate and diagnose something nobody set an alarm for. One warns about known dangers; the other lets you explore the unknown.
Step by Step
Key Concepts
The Three Pillars
Metrics (numeric trends over time), logs (detailed event records), and traces (a request path across services). Together they let you both detect and diagnose problems in distributed systems.
Observability vs Monitoring
Monitoring watches for predefined, known failure conditions. Observability provides rich enough data to investigate new, unforeseen problems — the "unknown unknowns" — after they appear.
SLOs, SLIs, and Error Budgets
An SLI is a measured indicator (e.g., success rate); an SLO is the target for it (99.9%); the error budget is the allowed shortfall. They turn observability data into reliability goals.
Structured Logging
Emitting logs as machine-parseable key-value data (JSON) rather than free text, so they can be filtered, aggregated, and correlated with metrics and traces at scale.
Key Facts
- Metrics tell you something is wrong, traces tell you where, and logs tell you why — you usually need all three to diagnose a real incident.
- Observability shines for "unknown unknowns" — novel failures in complex systems that no one thought to set an alert for in advance.
- OpenTelemetry has become the vendor-neutral standard for generating metrics, logs, and traces, so you can instrument once and switch backends.
Real-World Applications
Diagnosing a latency spike
A latency-alert metric leads an engineer to a slow trace, which pinpoints a single downstream service; that service structured logs reveal a slow query — a root cause found in minutes, not hours.
Reliability with SLOs
A team defines an SLO (99.9% of requests under 300ms), measures it from metrics, and spends its error budget deliberately — shipping faster when reliability is healthy and slowing down when it is spent.
Frequently Asked Questions
What are the three pillars of observability?
They are metrics, logs, and traces. Metrics are numeric measurements aggregated over time, ideal for dashboards and alerting. Logs are timestamped records of discrete events that provide detailed context. Traces follow a single request across all the services it touches, showing where time is spent and where failures occur. Combining all three lets you both detect and diagnose problems.
What is the difference between observability and monitoring?
Monitoring watches for predefined, known problems using dashboards and alerts — it answers questions you decided to ask in advance, like "is error rate above 5%?". Observability provides rich, high-cardinality telemetry that lets you ask new, unanticipated questions after a problem appears, so you can investigate novel failures ("unknown unknowns"). Monitoring is a subset of what good observability enables.
How do metrics, logs, and traces work together?
They complement each other. Metrics reveal that something is wrong (a latency or error-rate spike) and are great for alerting. A trace shows where in the distributed request the problem occurred, narrowing it to a specific service. That service logs then explain why, with the detailed context and values. Correlating the three — often via a shared trace ID — turns a vague symptom into a precise root cause.
What are SLOs and error budgets?
An SLI (Service Level Indicator) is a measured signal of reliability, such as the percentage of successful requests. An SLO (Service Level Objective) is the target for that indicator, like 99.9%. The error budget is the allowed amount of failure (the remaining 0.1%). Teams use observability data to track SLOs and spend the error budget deliberately — shipping features faster when reliability is healthy and prioritising stability when the budget is running low.