Home/Learn/Microservices/Graceful Shutdown

Graceful Shutdown

Intermediate

Patterns

On SIGTERM, stop accepting new requests, drain in-flight requests within a configurable timeout, then close connections — Spring Boot handles this via server.shutdown=graceful.

Overview

Graceful shutdown ensures that when a service receives a termination signal (SIGTERM from Kubernetes during a rolling deploy or scale-down), it finishes in-flight work before exiting — rather than abruptly closing connections and leaving clients with 500 errors. The correct sequence: (1) stop accepting new requests (deregister from load balancer), (2) wait for in-flight HTTP requests to complete, (3) drain message consumers (ack pending messages), (4) flush caches / close DB connections, (5) exit. Spring Boot handles steps 1–2 via `server.shutdown=graceful`. Kubernetes must be configured with a `preStop` hook and `terminationGracePeriodSeconds` to allow the drain window.

Spring Boot Graceful Shutdown Config

`server.shutdown=graceful` puts the embedded server into graceful shutdown mode: it stops accepting new requests immediately and waits up to `spring.lifecycle.timeout-per-shutdown-phase` for in-flight requests to complete before the application context is closed. Set this to slightly less than Kubernetes' `terminationGracePeriodSeconds`.

Spring Boot — graceful shutdown + Kubernetes config

# application.properties
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=20s   # wait up to 20s for in-flight HTTP requests

# --- Kubernetes Deployment YAML ---
spec:
  containers:
    - name: order-service
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 5"]    # wait 5s after deregister before SIGTERM
  terminationGracePeriodSeconds: 30              # total time before SIGKILL
  # Timeline:
  # t=0:  SIGTERM sent + preStop hook runs (5s sleep)
  # t=5:  Spring receives SIGTERM → stops accepting new requests
  # t=25: Drain window closes (20s)
  # t=30: SIGKILL if process still running

Kafka Consumer Graceful Shutdown

Kafka consumers must be drained before shutdown: finish processing the current message batch, commit offsets, and leave the consumer group cleanly. Spring Kafka's `ConcurrentMessageListenerContainer` stops the container on `SmartLifecycle.stop()`, which is called during Spring context shutdown. Register a `@PreDestroy` or rely on Spring's `SmartLifecycle` ordering.

Spring Kafka — SmartLifecycle consumer drain

@Component
class OrderConsumerManager implements SmartLifecycle {
    private final ConcurrentMessageListenerContainer<?, ?> container;
    private volatile boolean running = false;

    @Override
    public void start() {
        container.start();
        running = true;
    }

    @Override
    public void stop(Runnable callback) {
        // Called during graceful shutdown — Spring respects lifecycle.timeout-per-shutdown-phase
        log.info("Stopping Kafka consumer — waiting for in-flight messages...");
        container.stop(() -> {
            log.info("Kafka consumer stopped cleanly");
            running = false;
            callback.run();   // signal Spring: this lifecycle phase is complete
        });
    }

    @Override public boolean isRunning() { return running; }
    @Override public int getPhase() { return Integer.MAX_VALUE - 10; }  // shutdown late
}

# Spring Kafka also respects server.shutdown=graceful automatically via SmartLifecycle

Kubernetes Rolling Deploy and Readiness Probe

Graceful shutdown works in concert with Kubernetes' **readiness probe**: when a pod is shutting down, Kubernetes marks it not-ready so the load balancer stops routing traffic before the drain window begins. The `preStop` sleep gives the load balancer time to update its routing table — preventing new requests from arriving after SIGTERM. Without this sleep, requests can still arrive for a few seconds after SIGTERM.

Kubernetes — readiness probe + preStop for graceful rolling deploy

# Kubernetes Deployment — full graceful shutdown setup
spec:
  containers:
    - name: order-service
      readinessProbe:
        httpGet:
          path: /actuator/health/readiness
          port: 8080
        failureThreshold: 1
        periodSeconds: 5
      livenessProbe:
        httpGet:
          path: /actuator/health/liveness
          port: 8080
        failureThreshold: 3
        periodSeconds: 10
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]  # let LB drain before SIGTERM

  terminationGracePeriodSeconds: 60   # total budget: preStop(10) + drain(20) + buffer(30)

# application.properties
management.endpoint.health.probes.enabled=true
management.health.livenessstate.enabled=true
management.health.readinessstate.enabled=true

Key Points to Remember

1server.shutdown=graceful stops accepting new requests and waits for in-flight requests to finish
2spring.lifecycle.timeout-per-shutdown-phase controls the drain window (default: 30s)
3Kubernetes preStop sleep gives the load balancer time to deregister the pod before SIGTERM
4terminationGracePeriodSeconds = preStop time + drain window + buffer (set to their sum)
5Kafka consumers drain via SmartLifecycle.stop() — called automatically during context shutdown
6Readiness probe /actuator/health/readiness goes DOWN on shutdown — prevents new traffic routing

Interview Questions

What is the difference between graceful shutdown and a hard kill in Kubernetes?

EasyThoughtWorks

Why do you need a preStop sleep even with server.shutdown=graceful?

HardNetflix

How does the readiness probe work in conjunction with graceful shutdown?

MediumAmazon

What happens to a Kafka consumer's in-flight messages during a graceful shutdown?

MediumConfluent

How would you calculate the correct terminationGracePeriodSeconds value?

MediumGoogle

Ask Aria about Graceful Shutdown

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Backward Compatibility & Contract Testing

Microservices Anti-Patterns