Home/Learn/System Design/Health Checks & Heartbeats

Health Checks & Heartbeats

Beginner

Reliability & Fault Tolerance

Health checks probe services to determine if they are alive and ready to handle traffic. Heartbeats are periodic signals nodes send to indicate they are operational. Both are essential for automated failure detection.

Overview

Health checks and heartbeats are mechanisms for detecting failures in distributed systems. A health check is a probe sent by an external system (load balancer, Kubernetes, monitoring) to a service endpoint. The service responds with its status — healthy or unhealthy. Liveness checks determine if a process is alive (restart if not). Readiness checks determine if a service is ready to accept traffic (remove from LB if not). A heartbeat is a periodic signal a node sends to a coordinator to indicate it is alive. If heartbeats stop arriving, the coordinator assumes the node has failed. Health checks are pull-based (checker probes the service); heartbeats are push-based (service signals the coordinator). Both feed into automated remediation: restarting containers, removing from load balancer pools, triggering failover, or alerting operators.

Liveness vs Readiness Checks

Liveness checks detect crashed processes that need restarting. Readiness checks detect services that are alive but not ready to serve (still warming up, DB connection not established).

Java + YAML — Spring Boot health checks

// Spring Boot Actuator health endpoint
// GET /actuator/health → { "status": "UP" }
// GET /actuator/health/liveness → { "status": "UP" }
// GET /actuator/health/readiness → { "status": "UP" }

// application.yml
management:
  endpoint:
    health:
      show-details: when_authorized
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

// Custom health indicator
@Component
public class DatabaseHealthIndicator implements HealthIndicator {
    private final DataSource dataSource;

    @Override
    public Health health() {
        try (Connection conn = dataSource.getConnection()) {
            conn.createStatement().execute("SELECT 1");
            return Health.up().withDetail("database", "reachable").build();
        } catch (Exception e) {
            return Health.down().withException(e).build();
        }
    }
}

Kubernetes Health Probes

Kubernetes uses three types of probes: livenessProbe (restart pod if fails), readinessProbe (remove from Service endpoints if fails), and startupProbe (delay other probes until app is started).

Kubernetes YAML — liveness, readiness, startup probes

// Kubernetes pod with all three probes
apiVersion: v1
kind: Pod
metadata:
  name: order-service
spec:
  containers:
    - name: order-service
      image: order-service:1.0
      ports:
        - containerPort: 8080

      # Restart pod if liveness fails
      livenessProbe:
        httpGet:
          path: /actuator/health/liveness
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10
        failureThreshold: 3    # restart after 3 consecutive failures

      # Remove from load balancer if readiness fails
      readinessProbe:
        httpGet:
          path: /actuator/health/readiness
          port: 8080
        periodSeconds: 5
        failureThreshold: 2    # stop sending traffic after 2 failures

      # Delay liveness/readiness until startup succeeds
      startupProbe:
        httpGet:
          path: /actuator/health
          port: 8080
        failureThreshold: 30
        periodSeconds: 2       # up to 60s for slow-starting apps

Key Points to Remember

1Liveness = "is the process alive?" (restart if dead); readiness = "can it handle traffic?" (remove from LB if not).
2Health check endpoints should verify downstream dependencies (DB, cache, queues).
3Kubernetes probes: livenessProbe (restart), readinessProbe (traffic), startupProbe (slow startup).
4Heartbeats are push-based — the service periodically signals a coordinator that it is alive.
5Set appropriate timeouts and thresholds to avoid false positives (premature restarts).

Interview Questions

What is the difference between liveness and readiness health checks?

EasyTCS

How does Kubernetes use health probes to manage pods?

MediumAmazon

What should a health check endpoint verify?

MediumGoogle

How would you detect a node failure in a distributed system using heartbeats?

MediumFlipkart

Design a health-check system for a microservices platform with 200 services.

HardNetflix

Ask Aria about Health Checks & Heartbeats

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Failover & Redundancy

Graceful Degradation