Observability — Metrics, Logs & Tracing
AdvancedProduction Kubernetes observability uses Prometheus + Grafana for metrics, Loki or EFK for logs, and Jaeger or Tempo for distributed tracing. Together they form the three pillars of observability.
Overview
You cannot manage what you cannot observe. The three pillars of observability are: Metrics (numeric time-series — CPU, latency, error rate, queue depth), Logs (discrete events from applications and system components), and Traces (the journey of a request across multiple services). In Kubernetes, Prometheus scrapes metrics from pods (via /metrics endpoint and ServiceMonitors), Grafana visualises them, Loki or EFK aggregates logs, and OpenTelemetry + Jaeger/Tempo handles distributed tracing. The golden signals (latency, traffic, errors, saturation — LTES) from Google SRE are the four metrics that matter most.
Prometheus & Grafana Stack
Prometheus scrapes metrics from targets. kube-prometheus-stack (helm chart) deploys Prometheus, Grafana, AlertManager, and pre-built Kubernetes dashboards in one command.
# Install kube-prometheus-stack (Prometheus + Grafana + AlertManager)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace
# What it installs:
# - Prometheus Operator → manages Prometheus instances via CRDs
# - Prometheus → scrapes metrics
# - Grafana → dashboards (pre-built K8s dashboards)
# - AlertManager → routes alerts to Slack/PagerDuty
# - node-exporter → host-level metrics (CPU, memory, disk, network)
# - kube-state-metrics → K8s object metrics (pod counts, deployment status)
# Expose your app metrics via /metrics endpoint
# Add ServiceMonitor to tell Prometheus to scrape it:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-metrics
labels:
release: prometheus # must match Prometheus's serviceMonitorSelector
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: http
path: /metrics
interval: 15s # scrape every 15s
# Example Prometheus alert rule:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: my-app-alerts
spec:
groups:
- name: my-app
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
for: 5m
annotations:
summary: "Error rate > 1% for 5 minutes"Logging with Loki
Grafana Loki is like Prometheus but for logs. Promtail collects logs from all pods and ships them to Loki. Logs are queryable in Grafana alongside metrics — no separate logging dashboard needed.
# Install Loki stack (Loki + Promtail)
helm install loki grafana/loki-stack \
--namespace monitoring \
--set grafana.enabled=false # already installed with kube-prometheus-stack
# Promtail DaemonSet: runs on every node, reads /var/log/containers/*
# Automatically adds K8s labels: namespace, pod, container, node
# Query Loki in Grafana using LogQL:
# {namespace="production", app="my-app"} |= "ERROR"
# {app="my-app"} | json | level="error" | line_format "{{.message}}"
# Rate of error logs over time (mix metrics and logs):
# rate({app="my-app"} |= "ERROR" [5m])
# Alert on log pattern:
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: loki-alert
spec:
route:
receiver: slack
receivers:
- name: slack
slackConfigs:
- apiURL: ...
channel: '#alerts'
# Access Grafana
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
# Default: admin/prom-operatorDistributed Tracing with OpenTelemetry
Distributed tracing follows a request across microservices — showing which service was slow and why. OpenTelemetry is the standard instrumentation library; Jaeger or Tempo stores and queries traces.
# Install Jaeger (all-in-one for dev, distributed for prod)
helm install jaeger jaegertracing/jaeger \
--namespace monitoring
# Or use Grafana Tempo (integrates with Grafana dashboards)
helm install tempo grafana/tempo \
--namespace monitoring
# Instrument your Node.js app with OpenTelemetry
npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
// tracing.js — initialize at app startup
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-otlp-http')
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://jaeger-collector.monitoring:4318/v1/traces',
}),
instrumentations: [getNodeAutoInstrumentations()],
})
sdk.start()
// Auto-instruments: HTTP, Express, gRPC, PostgreSQL, Redis — no code changes!
# The Golden Signals (what to alert on):
# Latency: p99 of request duration > SLO threshold
# Traffic: requests per second (know your baseline)
# Errors: error rate (5xx / total) > threshold
# Saturation: CPU/memory/queue > 80% — leading indicator of problemsKey Points to Remember
- 1Three pillars of observability: Metrics (Prometheus), Logs (Loki/EFK), Traces (Jaeger/Tempo).
- 2kube-prometheus-stack deploys the full metrics stack in one helm install command.
- 3ServiceMonitor CRD tells Prometheus which pods to scrape — label matching is required.
- 4Loki + Promtail indexes log labels (namespace, pod, app) for fast filtered queries.
- 5OpenTelemetry auto-instrumentation requires zero code changes for most frameworks.
- 6Golden signals: Latency, Traffic, Errors, Saturation — alert on these four, not on every symptom.
Interview Questions
Sign in to ask AriaWhat are the three pillars of observability and how do they complement each other?
How does Prometheus discover which pods to scrape?
Ask Aria about Observability — Metrics, Logs & Tracing
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.