How Prometheus Works
IntermediatePrometheus is an open-source monitoring system built around a time-series database. Unlike tools that receive pushed metrics, Prometheus pulls (scrapes) metrics from your services over HTTP at regular intervals. It stores each measurement as a time series identified by a name and labels, lets you query and aggregate them with PromQL, and fires alerts through Alertmanager. Paired with Grafana for dashboards, it is a de facto standard for cloud-native metrics.
Think of a nurse doing hourly rounds
Rather than every patient calling out their vitals whenever they change (push), a nurse walks the ward on a schedule and reads each patient monitor (pull/scrape), writing every reading into a chart with the time and the patient name (a labelled time series). Later, doctors flip through the charts to spot trends (PromQL), and a rule triggers an alarm if any reading crosses a danger line (Alertmanager). Regular rounds, consistent records, and rules on top.
Step by Step
Key Concepts
Pull-based Scraping
Prometheus fetches metrics from targets on a schedule rather than receiving pushed data. It controls timing and detects down targets via failed scrapes; short-lived jobs use a push gateway as an exception.
Time Series and Labels
Every metric is a time series keyed by a name and a set of labels. Labels add dimensions (endpoint, status, instance), so one metric name yields many series you can filter and group.
Metric Types
Counter (monotonically increasing, e.g., total requests), Gauge (goes up and down, e.g., memory), Histogram and Summary (distributions, e.g., request-duration buckets for percentiles).
PromQL
The query language for selecting and aggregating time series — computing rates, percentiles, and cross-service aggregates — used for dashboards, alerts, and ad-hoc debugging.
Key Facts
- Prometheus pulls metrics, so a target that stops responding is itself a signal (the scrape fails) — you learn immediately that a service is unreachable.
- Labels are powerful but every unique label combination is a new time series; high-cardinality labels (like user IDs) can explode storage.
- Prometheus is typically paired with Grafana for visualisation and Alertmanager for routing alerts — it focuses on collection, querying, and alerting.
Real-World Applications
Service dashboards and SLOs
A team exposes request-rate, error, and latency metrics; PromQL computes the error rate and p95 latency, Grafana charts them, and alert rules fire when an SLO threshold is breached.
Infrastructure monitoring
Exporters expose host and database metrics (CPU, disk, connections) that Prometheus scrapes, giving a unified view of both application and infrastructure health in one system.
Frequently Asked Questions
How does Prometheus collect metrics?
Prometheus uses a pull model: it scrapes metrics from your services over HTTP at a configured interval, rather than having services push metrics to it. Each service exposes a /metrics endpoint (via a client library or an exporter), and Prometheus fetches it on schedule. This gives Prometheus control over timing and lets it detect when a target is down, because the scrape simply fails.
What is PromQL?
PromQL (Prometheus Query Language) is the language for selecting and aggregating the time-series data Prometheus stores. It lets you compute things like the per-second request rate, 95th-percentile latency, or total errors grouped by service. PromQL powers ad-hoc investigation, Grafana dashboard panels, and the conditions in alerting rules.
What are the Prometheus metric types?
There are four. A Counter only increases and is used for cumulative totals like requests served (you take its rate). A Gauge can go up or down, for values like memory usage or queue length. Histograms and Summaries record distributions of values — such as request durations across buckets — enabling percentile calculations like p95 and p99.
How does alerting work in Prometheus?
You define alerting rules as PromQL expressions with a condition and duration (for example, error rate above 5% for 5 minutes). When a rule condition holds, Prometheus fires an alert to Alertmanager, which deduplicates, groups, silences, and routes the alerts to destinations like Slack, email, or PagerDuty. This separation keeps rule evaluation in Prometheus and notification logic in Alertmanager.