Home/Learn/System Design/Monitoring & Alerting

Monitoring & Alerting

Intermediate

Observability & Operations

Monitoring collects metrics (CPU, latency, error rates, business KPIs) and alerting notifies teams when thresholds are breached. Prometheus + Grafana is the standard open-source stack.

Overview

Monitoring is the process of collecting, aggregating, and visualising metrics to understand system health and performance. Key metric types are: (1) Infrastructure metrics — CPU, memory, disk, network. (2) Application metrics — request rate, error rate, latency percentiles (p50, p95, p99). (3) Business metrics — orders/minute, revenue/hour, active users. The RED method (Rate, Errors, Duration) covers essential service health metrics. The USE method (Utilisation, Saturation, Errors) covers infrastructure. Prometheus scrapes metrics from services at regular intervals, stores them in a time-series database, and Grafana visualises them in dashboards. Alerting rules trigger notifications (PagerDuty, Slack, email) when metrics breach thresholds. Good alerts are actionable, have runbooks, avoid alert fatigue, and use severity levels (P1-critical, P2-warning, P3-info).

The RED & USE Methods

RED (Rate, Errors, Duration) monitors service health. USE (Utilisation, Saturation, Errors) monitors infrastructure. Together, they cover the most important metrics.

Conceptual + Java + YAML — RED/USE methods with Micrometer

// RED method — for every service
// Rate:     requests per second
// Errors:   error rate (5xx / total requests)
// Duration: latency percentiles (p50, p95, p99)

// USE method — for every resource (CPU, memory, disk, network)
// Utilisation: % of resource in use (CPU at 70%)
// Saturation:  work queued (thread pool queue depth)
// Errors:      error count (disk I/O errors)

// Spring Boot + Micrometer metrics (auto-exposed for Prometheus)
// application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,info
  metrics:
    tags:
      application: order-service
    distribution:
      percentiles-histogram:
        http.server.requests: true  # exposes p50, p95, p99

// Custom business metric
@Component
public class OrderMetrics {
    private final Counter ordersPlaced;
    private final Timer orderProcessingTime;

    public OrderMetrics(MeterRegistry registry) {
        this.ordersPlaced = Counter.builder("orders.placed")
            .tag("payment_method", "card")
            .register(registry);
        this.orderProcessingTime = Timer.builder("orders.processing.time")
            .register(registry);
    }
}

Prometheus + Grafana + Alerting

Prometheus scrapes metrics, stores time-series data, and evaluates alert rules. Grafana visualises metrics in dashboards. Alertmanager routes alerts to the right team.

YAML + PromQL — Prometheus config and alert rules

// Prometheus scrape config
scrape_configs:
  - job_name: 'order-service'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: order-service
        action: keep

// PromQL — query language
// Request rate per second
rate(http_server_requests_seconds_count[5m])

// Error rate (5xx)
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_count[5m]))

// p99 latency
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))

// Alert rule
groups:
  - name: order-service-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{status=~"5..", app="order-service"}[5m]))
          / sum(rate(http_server_requests_seconds_count{app="order-service"}[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Order service error rate above 5%"
          runbook: "https://wiki.example.com/runbooks/order-service-errors"

Key Points to Remember

1RED (Rate, Errors, Duration) for service health; USE (Utilisation, Saturation, Errors) for infrastructure.
2Prometheus scrapes metrics; Grafana visualises; Alertmanager routes alerts.
3Track latency percentiles (p50, p95, p99) — averages hide tail latency problems.
4Good alerts are actionable, have runbooks, use severity levels, and avoid alert fatigue.
5Business metrics (orders/min, revenue/hour) are as important as infrastructure metrics.

Interview Questions

What metrics would you monitor for a REST API service?

EasyTCS

What is the RED method for service monitoring?

MediumAmazon

Why is p99 latency more important than average latency?

MediumGoogle

How do you avoid alert fatigue in a large microservices system?

MediumFlipkart

Design the monitoring and alerting platform for a 200-service microservices architecture.

HardNetflix

Ask Aria about Monitoring & Alerting

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Logging Best Practices

Distributed Tracing