Home/Learn/Microservices/Timeouts & Deadline Propagation

Timeouts & Deadline Propagation

Intermediate

Resilience

Always set read/connect timeouts on HTTP clients; propagate deadlines through call chains so that downstream work is abandoned when the upstream caller has given up.

Overview

Timeouts are one of the most important resilience mechanisms in microservices — without them, a slow downstream service will eventually exhaust the thread pool of the calling service, causing cascading failures. There are two distinct timeout types: connect timeout (how long to wait for TCP handshake) and read timeout (how long to wait for a response after the request is sent). Deadline propagation extends this to distributed call chains: when Service A calls B which calls C, the deadline (remaining allowed time) should be forwarded so that C abandons work when A has already timed out. In HTTP this is often done via a custom X-Deadline header; in gRPC it is built in. Spring's RestClient, WebClient, and Feign all support per-client timeout configuration.

Configuring connect and read timeouts

Connect timeout should be short (1–3 s) — if TCP does not establish within that time the target is likely down. Read timeout should reflect the p99 latency of the downstream service under normal load plus a safety margin, not an arbitrary large number. Setting read timeout to 30 s when the service normally responds in 200 ms means threads will pile up for 30 s during an outage.

Java — RestClient, Feign, and WebClient timeout configuration

// Spring Boot 3 — RestClient with Apache HttpClient 5 timeouts
@Bean
public RestClient inventoryClient() {
    CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionRequestTimeout(Timeout.ofSeconds(2))   // connect timeout
        .setResponseTimeout(Timeout.ofSeconds(5))            // read timeout
        .build();

    return RestClient.builder()
        .baseUrl("http://inventory-service")
        .requestFactory(new HttpComponentsClientHttpRequestFactory(httpClient))
        .build();
}

// Feign client — application.yml
// spring:
//   cloud:
//     openfeign:
//       client:
//         config:
//           inventory-service:
//             connect-timeout: 2000   # ms
//             read-timeout: 5000      # ms

// WebClient with Netty reactor timeout
@Bean
public WebClient webClient() {
    HttpClient nettyClient = HttpClient.create()
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2_000)
        .responseTimeout(Duration.ofSeconds(5));
    return WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(nettyClient))
        .build();
}

Deadline propagation across service calls

A deadline is an absolute point in time by which the entire operation must complete. Unlike a timeout (which resets at each hop), a deadline shrinks as it passes through the call chain. When making downstream calls, subtract elapsed time from the remaining budget. gRPC propagates deadlines automatically; for HTTP services, use a custom header like X-Request-Deadline and enforce it with a filter.

Java — deadline propagation via HTTP header and thread-local context

// Deadline propagation via HTTP header
@Component
public class DeadlineFilter extends OncePerRequestFilter {

    @Override
    protected void doFilterInternal(HttpServletRequest req,
                                    HttpServletResponse res,
                                    FilterChain chain) throws ServletException, IOException {
        String deadlineHeader = req.getHeader("X-Request-Deadline");
        if (deadlineHeader != null) {
            Instant deadline = Instant.parse(deadlineHeader);
            if (Instant.now().isAfter(deadline)) {
                res.sendError(408, "Request deadline exceeded");
                return;
            }
            DeadlineContext.set(deadline);
        }
        try {
            chain.doFilter(req, res);
        } finally {
            DeadlineContext.clear();
        }
    }
}

// Forward remaining deadline when calling downstream services
public InventoryResponse getStock(String productId) {
    Instant deadline = DeadlineContext.get();
    return inventoryClient.get()
        .uri("/stock/{id}", productId)
        .header("X-Request-Deadline", deadline.toString())
        .retrieve()
        .body(InventoryResponse.class);
}

Timeout budgeting and circuit breaker combination

A common mistake is setting each downstream timeout to the full budget (e.g., 10 s), which means a call chain of A→B→C can take up to 30 s. Each service should budget timeouts proportionally so the total fits within the upstream SLA. Combine timeouts with circuit breakers: if a service consistently times out, trip the circuit to fail fast instead of waiting each time.

Java — Resilience4j TimeLimiter + CircuitBreaker with fallback

// Resilience4j — combine timeout + circuit breaker
@Bean
public TimeLimiterConfig timeLimiterConfig() {
    return TimeLimiterConfig.custom()
        .timeoutDuration(Duration.ofSeconds(3))
        .build();
}

@Bean
public CircuitBreakerConfig circuitBreakerConfig() {
    return CircuitBreakerConfig.custom()
        .slidingWindowSize(10)
        .failureRateThreshold(50)
        .waitDurationInOpenState(Duration.ofSeconds(15))
        .recordExceptions(TimeoutException.class, ConnectException.class)
        .build();
}

// Annotation stack: TimeLimiter wraps the call, CircuitBreaker counts failures
@CircuitBreaker(name = "inventory", fallbackMethod = "fallbackStock")
@TimeLimiter(name = "inventory")
public CompletableFuture<Integer> getStockAsync(String productId) {
    return CompletableFuture.supplyAsync(
        () -> inventoryClient.getStock(productId));
}

public CompletableFuture<Integer> fallbackStock(String productId, Throwable t) {
    log.warn("Inventory unavailable for {}: {}", productId, t.getMessage());
    return CompletableFuture.completedFuture(0);
}

Key Points to Remember

1Always set both connect timeout (TCP establishment) and read timeout (response wait) on every HTTP client
2Base read timeouts on actual p99 latency + safety margin — not arbitrary large values like 30 s
3Deadlines are absolute timestamps that shrink across call chains; timeouts reset at each hop
4gRPC propagates deadlines automatically; HTTP services need explicit header forwarding (e.g. X-Request-Deadline)
5Combine timeouts with circuit breakers — timeouts prevent individual call hangs, circuit breakers prevent repeated waits
6Timeout budget each leg proportionally: in A→B→C with a 5 s SLA, each leg timeout must fit within 5 s total

Interview Questions

What is the difference between a timeout and a deadline in distributed systems?

MediumGoogle

Why is a read timeout of 30 s dangerous for a service that normally responds in 200 ms?

EasyAmazon

How would you propagate deadlines across a chain of HTTP microservice calls?

HardUber

How does Resilience4j TimeLimiter differ from an HTTP client read timeout?

MediumThoughtworks

Describe a timeout budget strategy for a 3-tier service call chain with a 5 s end-to-end SLA.

HardNetflix

Ask Aria about Timeouts & Deadline Propagation

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

Bulkhead Pattern

Database per Service Pattern