Microservices — Cheat Sheet

Microservices is an architectural style where an application is built as a collection of small, independently deployable services, each owning its domain and data. Problems solved vs monolith: • Independent deployability: Deploy one service without redeploying everything • Technology flexibility: Each service can use the right language/DB for its problem • Fault isolation: One service failure doesn't take down the whole application • Scalability: Scale only the services under load Tradeoffs introduced: • Distributed system complexity (network failures, latency, partial failure) • Data consistency is harder (no shared DB, cross-service transactions) • Operational overhead (many services to deploy, monitor, and secure) • Start with a modular monolith; extract services when team/scaling demands it.

An API Gateway is a single entry point for all client requests to a microservices backend. It routes requests to the appropriate service. Functions: • Request routing: Route /orders → Order Service, /users → User Service • Authentication: Validate JWT/OAuth tokens centrally (services don't need auth logic) • Rate limiting and throttling • SSL termination • Request/response transformation • Load balancing • Circuit breaking and retries • Aggregation: Combine responses from multiple services into one response (BFF pattern) Examples: AWS API Gateway, Kong, NGINX, Spring Cloud Gateway, Traefik. Downside: Single point of failure if not made highly available; adds one network hop.

Two main patterns: 1. Synchronous (Request-Response): • REST/HTTP: Simple, human-readable, widely supported. Adds latency chain (A → B → C must all be up). • gRPC: Binary protobuf, HTTP/2, streaming support. Faster and type-safe for internal service communication. 2. Asynchronous (Event-Driven): • Message queues (Kafka, RabbitMQ, SQS): Services publish events; consumers process independently. Decoupled in time and space. Better for high throughput and resilience. Choose synchronous when: Immediate response is required (payment validation), real-time UI updates. Choose async when: Action can be deferred (send email, update analytics), fan-out to multiple consumers, avoiding tight coupling.

Service discovery allows services to find each other dynamically without hardcoded IP addresses (which change in containerized environments). Two patterns: • Client-side discovery: The client queries a service registry (Eureka, Consul) and load-balances itself. Library-heavy (Netflix Ribbon). Clients need registry SDK. • Server-side discovery: Client sends request to load balancer; LB queries registry and routes. Client is unaware of registry. Used by AWS ELB, Kubernetes Services, Nginx. Kubernetes handles service discovery natively: Services get stable DNS names (my-service.namespace.svc.cluster.local) that route to healthy pods via kube-proxy, regardless of pod IP changes.

Cascading failures occur when a slow/failing downstream service causes upstream services to accumulate threads waiting on it, eventually exhausting thread pools and crashing the whole call chain. Circuit Breaker stops this by tracking failure rates and "tripping" when failures exceed a threshold: • Closed: Normal operation. Failures tracked. • Open: All calls fail fast (no network call). After a wait, try Half-Open. • Half-Open: Send a probe request. Success → Closed. Failure → back to Open. Implementation: Resilience4j in Java. Configure: failure rate threshold (50%), sliding window size, wait duration in Open state. Always combine with a fallback: return cached data, a default response, or a user-friendly error instead of propagating failure.

In microservices, a single user request spans multiple services. Traditional logs from individual services can't tell you the full story of a request's journey. Distributed tracing assigns a trace ID to the initial request. Each service adds a span with its processing time and propagates the trace ID in HTTP headers (or Kafka headers). This produces a trace: a tree of spans showing which service called which, how long each took, and where errors occurred. Tools: • Jaeger (CNCF) — open source, Kubernetes-native • Zipkin — lightweight, widely used • AWS X-Ray, Datadog APM, Dynatrace • OpenTelemetry — vendor-neutral standard for instrumentation Critical for: Identifying latency bottlenecks, tracing errors across services, understanding service dependencies.

The Strangler Fig pattern is a strategy for migrating from a monolith to microservices incrementally — without a risky "big bang" rewrite. Approach: 1. Route all traffic through a facade (API Gateway or reverse proxy) 2. Identify and extract a bounded context (e.g., User Service) 3. Implement the functionality in a new microservice 4. Route relevant requests from the facade to the new service 5. Once fully migrated, "strangle" (remove) the corresponding code from the monolith 6. Repeat for the next domain Benefits: Low risk, ship continuously, can pause migration at any stage. The monolith shrinks over time while the new architecture grows. Key challenge: Data — you may need to synchronize data between the monolith DB and the new service's DB during transition.

The sidecar pattern deploys a helper container (sidecar) alongside each service container in the same pod (Kubernetes). The sidecar handles infrastructure concerns so the main service doesn't have to. Common sidecar use cases: • Service mesh proxy (Envoy in Istio): Handles mTLS, retries, circuit breaking, traffic shaping, metrics — transparently, without changes to the service code • Log forwarding: Filebeat or Fluentd sidecars forward logs to Elasticsearch/Datadog • Config sync: Vault Agent sidecar syncs secrets from HashiCorp Vault • Network proxies: Linkerd or Consul Connect sidecars for secure service-to-service communication Benefit: Operational concerns (security, observability) are attached to every service uniformly without changing service code.

Options in order of maturity: 1. Environment variables: 12-factor app standard. Simple, works everywhere. No versioning. 2. ConfigMaps/Secrets (Kubernetes): Native K8s objects, injected as env vars or volumes. Secrets are base64-encoded (not encrypted at rest without KMS integration). 3. Spring Cloud Config Server: Centralized config server backed by a Git repo. Services fetch config on startup. Supports profiles and encryption. 4. HashiCorp Vault: Secrets management with dynamic secrets (short-lived DB credentials), encryption-as-a-service, access policies. 5. AWS Secrets Manager / Parameter Store: Cloud-native, integrates with IAM, rotates secrets automatically. Best practice: Never hardcode secrets, never commit secrets to Git, rotate credentials regularly.

The 12-Factor methodology defines best practices for cloud-native, scalable, and maintainable services: 1. Codebase: One codebase, many deploys (Git) 2. Dependencies: Explicitly declare (Maven, npm) 3. Config: Store in environment, not code 4. Backing services: Treat as attached resources (DB, cache) 5. Build, release, run: Strict separation of stages 6. Processes: Stateless processes, share nothing 7. Port binding: Export services via port 8. Concurrency: Scale out via processes 9. Disposability: Fast startup, graceful shutdown 10. Dev/prod parity: Keep environments similar 11. Logs: Treat as event streams (stdout → log aggregator) 12. Admin processes: Run management tasks as one-off processes These principles are foundational for containerized microservices and Kubernetes deployments.

Event-driven architecture (EDA) structures services around events — significant changes in state. Services publish events; other services consume and react to them without direct coupling. Components: Event producers, event broker (Kafka, RabbitMQ), event consumers. Benefits: • Loose coupling: Producers don't know consumers • Scalability: Add consumers without changing producers • Resilience: Consumers process at their own pace, broker absorbs bursts • Audit trail: Events are a natural history of what happened Challenges: Debugging distributed flows, ensuring idempotency, handling event schema evolution, eventual consistency. Use EDA for: Order processing pipelines, real-time analytics, cross-service data sync, notification systems, IoT data ingestion. Avoid for: Simple CRUD, low-latency user-facing reads.

Both are Saga implementations for distributed transactions, but they differ in who controls the flow. Choreography: No central coordinator. Each service listens for events, does its work, and publishes the next event. Decentralized, highly decoupled. Hard to visualize the overall flow; debugging requires tracing events across services. Orchestration: A central Saga Orchestrator service explicitly tells each participant what to do via commands and listens to their replies. Easier to understand (single place to see the whole flow), simpler error handling, but adds a central component that can become a bottleneck. Choose choreography for: Simple sagas, teams comfortable with event-driven patterns. Choose orchestration for: Complex sagas with many participants, where visibility into the process is important (Camunda, Temporal, or custom state machine).

Testing pyramid for microservices: 1. Unit tests: Test individual classes/functions in isolation with mocked dependencies. Fast, plentiful. 2. Component tests: Test one service in isolation with its real dependencies (real DB via Testcontainers) but mock other services via WireMock or MockServer. 3. Contract tests (Pact): Consumer defines the contract (expected API shape), provider verifies it. Detects API breaking changes without full integration tests. Faster and more reliable than end-to-end tests. 4. Integration tests: Test real service-to-service communication in a shared environment. Slower, flaky, but catches real integration bugs. 5. End-to-end tests: Full user journeys through the system. Use sparingly — very slow and brittle. Key tool: Testcontainers for spinning up real databases, Kafka, and Redis in tests.

A service mesh manages all service-to-service communication via sidecar proxies (Envoy), providing: • mTLS: Automatic mutual TLS encryption between all services • Observability: Metrics, traces, and logs for every service call without code changes • Traffic management: Canary deployments, A/B testing, weighted routing • Resilience: Retries, timeouts, circuit breaking at the infrastructure level Popular implementations: Istio (powerful, complex), Linkerd (lightweight, simpler), Consul Connect. When to use: When you have 10+ services and need consistent security, observability, and traffic control across all of them. The sidecar overhead (~10ms added latency) and operational complexity are only worth it at scale. For smaller deployments, application-level libraries (Resilience4j) and OpenTelemetry are often sufficient.

True ACID transactions across services are not possible without 2PC (which is slow and fragile). The alternative is eventual consistency with compensating actions. Strategies: • Saga pattern: Chain of local transactions + compensating transactions on failure • Outbox pattern: Atomic write to DB + outbox table, relay publishes events reliably • Event Sourcing: State is derived from an immutable event log — easy to reprocess and audit • Idempotency keys: Design every operation to be safe to retry (same request ID produces same result) • Deduplication: Consumers track processed event IDs (in a DB or Redis) to skip duplicates Accept eventual consistency where possible: Most business processes (order placed → inventory reserved → email sent) don't require instant consistency — they just need to eventually complete correctly.

Circuit breaker wraps calls to external services and monitors failures. When failures exceed a threshold, it "opens" — subsequent calls fail fast without attempting the remote call, preventing resource exhaustion. States: • CLOSED: Normal operation. Calls pass through. Track failure rate. • OPEN: Fail fast immediately. No calls to failing service. Timer starts. • HALF_OPEN: After timeout, allow a probe call. Success → close. Failure → reopen. Key parameters: failure rate threshold, minimum calls before evaluation, wait duration in open state. Resilience4j: ```java CircuitBreakerConfig config = CircuitBreakerConfig.custom() .failureRateThreshold(50) .waitDurationInOpenState(Duration.ofSeconds(30)) .build(); ``` Fallback options: Return cached response, default value, or a friendly error. Circuit breaker prevents cascading failures where one slow service takes down everything upstream.

Service discovery: The mechanism by which services find each other's network locations (IP + port) dynamically — without hardcoding addresses. Why needed: In dynamic environments (Kubernetes, auto-scaling), service instances start and stop constantly. IPs change. Service discovery provides a live registry. Client-side discovery: Client queries service registry (Consul, Eureka), gets a list of instances, and picks one using a load-balancing strategy (round-robin). Client has control but must include discovery logic. Server-side discovery: Client sends request to a load balancer or router (Kubernetes Service, AWS ALB). The router queries the registry and forwards the request. Client knows nothing about discovery. Kubernetes: Built-in DNS-based service discovery. Services get a stable DNS name (my-service.namespace.svc.cluster.local). Kubernetes kube-proxy maintains endpoints list. Pods communicate via Service DNS name — Kubernetes resolves to a healthy pod IP automatically. Spring Cloud with Eureka: Services register on startup, heartbeat to stay registered, deregister on shutdown. Ribbon (client-side) selects instances from the registry.

Bulkhead: Isolate resources (thread pools, connection pools) per external dependency so that a failure or slowdown in one dependency doesn't exhaust resources for others. Named after ship bulkheads — watertight compartments that prevent flooding from sinking the whole ship. Problem without bulkhead: Service A calls both Service B and Service C using the same thread pool. Service B starts responding slowly. Threads pile up waiting for B. All threads are consumed — now Service C calls also fail, even though C is healthy. Solution: Separate thread pools per dependency. Service B gets a pool of 10 threads. Service C gets a separate pool of 10 threads. Slowness in B only blocks B's 10 threads — C continues to function. Resilience4j Bulkhead: ```java BulkheadConfig config = BulkheadConfig.custom() .maxConcurrentCalls(10) .maxWaitDuration(Duration.ofMillis(100)) .build(); ``` Semaphore-based bulkhead: Limits concurrent calls without a separate thread pool — lighter weight but doesn't isolate thread context. Thread pool bulkhead: Executes calls in a separate thread pool — true isolation but more resource overhead.

Retry: Automatically re-attempt a failed operation, assuming the failure may be transient (network blip, temporary overload). When to retry: Transient errors — HTTP 429 (rate limited), 503 (service unavailable), connection timeout, read timeout. Do NOT retry on: 4xx client errors (400, 401, 403, 404), non-idempotent operations that already executed (POST payments), or when already in retry loop. Exponential backoff: Increase wait time between attempts exponentially: 100ms → 200ms → 400ms → 800ms. Prevents hammering a struggling service. Jitter: Add random offset to backoff: delay = base × 2^attempt + random(0, base). Prevents synchronized retry storms — multiple clients backing off at identical intervals then retrying simultaneously. Max attempts: Set a cap (3-5 attempts). Beyond that, fail and report. Resilience4j Retry: ```java RetryConfig config = RetryConfig.custom() .maxAttempts(3) .waitDuration(Duration.ofMillis(100)) .retryExceptions(IOException.class, TimeoutException.class) .ignoreExceptions(BusinessException.class) .build(); ``` Idempotency: Retries must only be applied to idempotent operations. Use idempotency keys for POST operations to make them retry-safe.

Both coordinate multi-step distributed transactions across microservices, but differ in how the coordination happens. Choreography: Services react to events without a central coordinator. Each service subscribes to events and publishes its own events when done. Example: OrderCreated event → Payment Service charges → PaymentSucceeded event → Inventory Service reserves → InventoryReserved event → Shipping service schedules. Pros: Loose coupling, no central SPOF, services are autonomous. Cons: Business flow is implicit — distributed across multiple services. Hard to see the whole picture. Risk of cyclic dependencies. Debugging requires tracing events across many services. Orchestration: A central Saga Orchestrator tells each participant what to do via commands and receives replies. Example: OrderSaga orchestrator sends ChargePayment to Payment Service → receives PaymentSucceeded → sends ReserveInventory to Inventory Service → etc. Pros: Business logic centralized in orchestrator — easy to understand and debug. Single place to handle all compensation logic. State machine visible in one place. Cons: Orchestrator is a central component. Risk of becoming a "God service" with too much logic. General recommendation: Orchestration for complex, multi-step business processes. Choreography for simple, linear event flows.

BFF: A separate API layer tailored specifically to the needs of a particular frontend client (mobile app, web app, third-party API). Problem: A generic API must serve all clients. Mobile needs minimal data (bandwidth constraints). Web needs rich aggregated data. Third-party APIs need different auth models. One API serving all becomes a compromise that satisfies none well. Solution: Create a dedicated BFF per client type: • Mobile BFF: Returns compressed, minimal JSON. Aggregates calls. Handles offline sync. • Web BFF: Returns rich, pre-joined data. Handles SSR data fetching. • Partner BFF: Stable, versioned, externally documented API. BFF responsibilities: • Aggregate multiple backend microservice calls into one client call • Transform/reshape data for the specific client's needs • Handle client-specific auth flows • Apply client-specific rate limits • Cache at the edge for the client's access patterns Ownership: Ideally owned by the frontend team — they control their own BFF and aren't blocked by backend teams. Not a replacement for microservices: BFF sits on top of microservices and orchestrates their calls. The core services remain generic and reusable.

Distributed tracing: Track a single request as it flows through multiple microservices, collecting timing data (spans) from each service. Concepts: • Trace: Entire journey of one request across all services. Has a unique trace-id. • Span: One unit of work within a trace (one service call, one DB query). Has span-id, parent-span-id, start/end times. • Context propagation: trace-id and span-id passed in HTTP headers (W3C TraceContext: traceparent header, or B3 headers for Zipkin compatibility). Implementation with OpenTelemetry (standard): 1. Add OpenTelemetry SDK to each service 2. Instrument HTTP clients and servers (auto-instrumentation via agents — zero code change) 3. Configure exporter: send spans to Jaeger, Zipkin, Grafana Tempo, or AWS X-Ray 4. Visualize: Trace waterfall shows exactly where time was spent across services Spring Boot: spring-boot-starter-actuator + Micrometer Tracing auto-propagates trace context across RestTemplate, WebClient, Feign, Kafka. What to trace: HTTP calls, DB queries, Kafka produces/consumes, cache hits/misses. Sampling: Don't trace 100% of requests in high-traffic systems (storage cost). Head-based sampling (decide upfront), or tail-based (keep only slow/error traces).

Strangler Fig: Incrementally replace parts of a legacy monolith with microservices, without a risky big-bang rewrite. The new system gradually "strangles" the old one. Named after the fig tree that wraps around and eventually replaces its host tree. Steps: 1. Place a facade/routing layer (reverse proxy or API gateway) in front of the monolith 2. Extract one bounded context at a time into a microservice 3. Route traffic for that context to the new service (the proxy handles routing) 4. The monolith still handles everything else 5. Repeat until the monolith handles nothing — decommission it Example: Start with the "Notifications" module — low risk, clear boundaries. Build Notification Service. Route /api/notifications to new service. Validate. Then extract "User Profile." Then "Orders." Data migration: Use the Strangler pattern for data too — new service has its own DB. Sync data from monolith DB to new DB via CDC during transition. Cut over reads when new DB is caught up. Benefits: No downtime. Easy rollback (reroute traffic to monolith). Incrementally validate each extracted service. Business continues while refactoring. Pitfall: The facade layer must not become a bottleneck or single point of failure — make it highly available.

Eventual consistency: In a distributed system, after a write, reads from different nodes may not immediately reflect the write — but given no new updates, all reads will eventually converge to the same value. In microservices: Service A updates its DB and publishes an event. Service B consumes the event and updates its own read model. Between the event publish and B's consumption, B's data is stale. Designing for it: • Identify which operations CAN be eventually consistent (inventory display, friend count, social feed) vs which CANNOT (payment deducted vs balance available, two users claiming the same seat). • Design UIs for it: Show "Updating..." states, disable submit buttons after action, show optimistic updates. • Idempotent consumers: Events may be delivered more than once. Use event_id to deduplicate. • Compensating transactions: If final state is wrong, business logic corrects it (refund, cancel, reconcile). • Monitoring: Track event processing lag. Alert on events stuck in queue. When strong consistency IS required: Use synchronous calls and explicit locks/transactions. Don't force eventual consistency for checkout inventory or financial balances. The key insight: Most business operations can tolerate some lag — the system doesn't need to be perfectly consistent at every millisecond, just correct eventually.

API versioning: Allows evolving your API without breaking existing callers. Critical in microservices because services are deployed independently — new and old versions coexist. Versioning strategies: • URL path: /api/v1/orders, /api/v2/orders. Most explicit, easy to test and document. Recommended for REST APIs. • Header: API-Version: 2 or Accept: application/vnd.company.v2+json. Clean URLs but invisible in browser, harder to test. • Query param: /api/orders?version=2. Easy to add but optional — easy to forget. Breaking vs non-breaking changes: • Non-breaking (backward-compatible): Adding new optional fields, adding new endpoints, relaxing required fields → safe without version bump • Breaking: Removing/renaming fields, changing field types, changing HTTP methods, changing response structure → requires new version Consumer-driven contracts (Pact): Providers and consumers agree on the contract. Consumers define what they expect. Providers verify they satisfy contracts before deploying. Prevents breaking changes from reaching production. Deprecation strategy: Mark old version deprecated (via Deprecation + Sunset response headers). Monitor usage metrics. Give consumers 6-12 months. Sunset old versions with prior announcement. Inter-service communication: For gRPC, use Protocol Buffer backward-compatibility rules (never remove field numbers, add new fields as optional).

Sidecar: Deploy a helper container/process alongside the main service container that augments or extends the service's capabilities — without modifying the service itself. Named after the motorcycle sidecar — a companion that shares the same lifecycle and resources as the primary vehicle. Common sidecars: • Service mesh proxy (Envoy/Linkerd): Intercepts all network traffic to/from the main service. Handles mTLS, retries, circuit breaking, load balancing, distributed tracing — transparently. • Log shipping: Filebeat/Fluentd reads logs written by the main container and ships them to ELK or Datadog. • Config/secrets refresher: Watches Vault or Kubernetes secrets and syncs changes to a file the main service reads. • Monitoring agent: Collects metrics from the main process and exposes them to Prometheus. Deployment: In Kubernetes, sidecars are containers in the same Pod. They share the network namespace (same localhost) and can share volumes. Benefits: • Cross-cutting concerns removed from application code • Language-agnostic — sidecar works with any language/runtime • Consistent behavior across all services (security, observability) enforced at infrastructure level Downside: Increased resource consumption (one sidecar per pod), more complexity in debugging (is the issue in the app or the sidecar?).

Dead letter queue (DLQ): A queue where messages that cannot be successfully processed are sent after exhausting retry attempts. A safety net for messages that would otherwise be silently dropped. When a message goes to DLQ: • Consumer throws an exception processing the message after N retries • Message has exceeded its TTL (time to live) • Message is malformed and can't be deserialized • Business logic explicitly rejects the message Why important: • Prevents message loss — failed messages aren't dropped, they're quarantined • Enables debugging — inspect DLQ to understand what failed and why • Allows replay — after fixing the bug, replay messages from DLQ back to the main queue • Prevents poison pills — a malformed message that crashes the consumer on every attempt would block the entire queue without a DLQ. With DLQ, it moves on and processing continues. SQS DLQ: Configure maxReceiveCount (e.g., 5). After 5 failed receives, message goes to DLQ. Alarm on DLQ message count. Kafka DLQ pattern: On processing exception, produce the message to a topic named original-topic-dlt (dead letter topic). Consumer group for DLT processes or stores for analysis. Operational: Always have monitoring/alerting on DLQ depth. DLQ messages growing = production bugs needing attention.

Health checks: Endpoints that report whether a service instance is ready to handle traffic. Used by load balancers, Kubernetes, and service registries to route only to healthy instances. Types: • Liveness probe: Is the service alive (not deadlocked/crashed)? If fails, restart the container. Simple check — can the service respond at all? • Readiness probe: Is the service ready to receive traffic? If fails, remove from load balancer rotation but don't restart. Checks dependencies: DB connected, cache reachable, initialization complete. • Startup probe: Is the service done starting up? Prevents liveness probe from killing a slow-starting container. Only active during startup. Spring Boot Actuator: /actuator/health endpoint with component checks. ``` { "status": "UP", "components": { "db": {"status": "UP"}, "redis": {"status": "UP"}, "diskSpace": {"status": "UP"} } } ``` Custom health indicators: Implement HealthIndicator to check domain-specific dependencies (Kafka connectivity, external API availability). Kubernetes configuration: ```yaml livenessProbe: httpGet: {path: /actuator/health/liveness, port: 8080} initialDelaySeconds: 30 readinessProbe: httpGet: {path: /actuator/health/readiness, port: 8080} initialDelaySeconds: 10 ``` Best practice: Liveness should be lightweight (no external calls). Readiness can check dependencies. Never let a health check cascade-fail (a DB outage shouldn't make all instances fail readiness simultaneously).

Outbox pattern: Solves the dual-write problem — writing to the DB and publishing an event reliably, without distributed transactions. Problem: Service writes to DB (success) then publishes to Kafka (fails) → event lost. Or: publishes to Kafka (success) then writes to DB (fails) → phantom event. Solution: 1. In the same DB transaction: Write business data AND write event to an outbox table. 2. A separate Message Relay process reads the outbox table and publishes events to Kafka. 3. Mark events as published (or delete them) after successful publish. Outbox table: { id, event_type, aggregate_id, payload JSON, created_at, published_at } Relay implementations: • Polling: Relay queries outbox for unpublished events on a schedule (simple, adds slight latency, puts query load on DB). • CDC/Debezium: Monitors DB transaction log (WAL/binlog). Picks up outbox table inserts in real-time with sub-second latency. No polling query. Recommended for production. Delivery guarantee: At-least-once (relay may retry on failure → duplicate events). Consumers must be idempotent (deduplicate using event_id). Cleanup: Delete or archive outbox rows after publication to prevent unbounded growth.

Idempotency: An operation that produces the same result regardless of how many times it's executed. f(f(x)) = f(x). Why critical in microservices: Network retries, message queue redelivery, and timeout-then-retry all cause duplicate requests. Without idempotency, a payment could be charged twice or an order created twice. Implementation patterns: 1. Idempotency key: Client generates a unique key (UUID) per logical operation. Server checks if this key was already processed. - Key exists in idempotency store → return stored response - Key not found → process, store result keyed by ID, return result Storage: Redis (with TTL) or idempotency_requests table. 2. Natural idempotency: Design operations so repeating them is harmless. - SET instead of INCREMENT - INSERT ... ON CONFLICT DO NOTHING - UPDATE ... WHERE status = 'pending' (only transitions once) 3. Event deduplication: Kafka or SQS may redeliver messages. Consumers track processed event_ids in Redis or DB. Skip already-seen events. 4. State machine transitions: An order can only transition PENDING → CONFIRMED once. Idempotent by design — second attempt sees it's already CONFIRMED and no-ops. Scope: Idempotency keys should have a TTL (24 hours typical) — don't store forever.

The 12-Factor App: A methodology for building scalable, maintainable software-as-a-service apps, especially relevant for microservices. 1. Codebase: One codebase per service, tracked in version control. Multiple deploys from one repo. 2. Dependencies: Explicitly declare all dependencies (pom.xml, package.json). No reliance on system-level packages. 3. Config: Store config in environment variables (not in code or files committed to repo). Config varies per environment; code doesn't. 4. Backing services: Treat DB, cache, message queue as attached resources — swappable via config. 5. Build, release, run: Strict separation. Build = compile. Release = build + config. Run = execute release. 6. Processes: Execute as one or more stateless processes. Persistent state in backing services, not process memory. 7. Port binding: Export services via port binding — service is self-contained, not deployed into a container like Tomcat. 8. Concurrency: Scale out via the process model (horizontal scaling). 9. Disposability: Fast startup, graceful shutdown. Handle SIGTERM. Enable rolling deployments. 10. Dev/prod parity: Keep development, staging, and production as similar as possible. 11. Logs: Treat logs as event streams — write to stdout, let infrastructure route/aggregate. 12. Admin processes: Run admin tasks (migrations, scripts) as one-off processes in the same environment.

Secret management: Securely providing credentials (DB passwords, API keys, TLS certs) to services without hardcoding them. Anti-patterns to avoid: • Secrets in code/config files committed to Git — never acceptable • Secrets in Docker images — any image pull leaks them • Secrets in environment variables (less severe but still risky — visible in process listings, logs) Production approaches: HashiCorp Vault: Dynamic secrets (generates DB credentials per service, auto-rotates). Fine-grained policies. Audit logging of every secret access. Sidecar agent injects secrets into pod. Kubernetes Secrets: Base64-encoded (not encrypted by default). Enable encryption-at-rest (AES-GCM with etcd encryption config). Use external-secrets-operator to sync from Vault or AWS Secrets Manager to K8s secrets. AWS Secrets Manager / Parameter Store: Managed service. IAM-based access control. Automatic rotation for RDS credentials. SDK reads secrets at runtime. Google Cloud Secret Manager / Azure Key Vault: Equivalent managed offerings. Best practices: • Never log secrets (filter in logging config) • Short-lived secrets with automatic rotation • Principle of least privilege — each service accesses only its own secrets • Audit all secret access • Vault approle or Kubernetes ServiceAccount for machine authentication to the secret store

Ambassador: A proxy service (or sidecar) that handles network communication on behalf of the application service. Acts as an "ambassador" that manages all the complexity of connecting to external services. Similar to Sidecar but specifically focused on network communication. Use cases: • Offload cross-cutting concerns: retry logic, circuit breaking, mTLS, timeout, logging of external calls — from the application to the ambassador • Protocol translation: Application speaks HTTP; ambassador translates to gRPC or legacy SOAP • Authentication proxy: Ambassador handles service-to-service auth (mTLS certificates, token injection) so application code stays clean • Rate limit outbound calls: Ambassador throttles calls from the service to external APIs Deployment: As a sidecar in the same pod (shares localhost). Outbound calls go to localhost:XXXX where ambassador listens, then ambassador makes the actual external call. Example: Service mesh Envoy proxy is essentially an ambassador — it handles all outbound traffic with retries, circuit breaking, and tracing without the app knowing. Vs Sidecar: Sidecar is the general pattern (any helper co-process). Ambassador is a specialization focused on network communication proxy. Vs API Gateway: Gateway handles inbound traffic from external clients. Ambassador handles outbound traffic from a service to its dependencies.

Service-to-service authentication: Verify that the calling service is who it claims to be — not just that the request has a valid user JWT. Approaches: 1. mTLS (Mutual TLS): Both client and server present TLS certificates during handshake. Server verifies client cert (signed by trusted CA), client verifies server cert. Strong cryptographic identity. Managed by service mesh (Istio/Linkerd auto-rotates certs). Zero application code. 2. JWT with service identity: Service A calls the identity provider with its service credentials → gets a short-lived JWT with service identity claims. Attaches JWT to outbound requests. Service B validates the JWT signature and checks the issuer/subject claims. 3. API keys: Service A presents a secret API key in a header. Service B validates the key against a registry. Simple but keys can be compromised, rotation is manual. 4. Kubernetes ServiceAccount + OIDC: Kubernetes mints JWTs for each ServiceAccount. Services authenticate to other services or external systems (AWS, Vault) using their ServiceAccount JWT. 5. OAuth2 Client Credentials flow: Service A authenticates to the auth server with client_id + client_secret → gets an access token. Uses the token for service B calls. B validates the token at the auth server or via JWT signature. Best practice: Use mTLS at the network layer (service mesh) combined with JWT at the application layer for defense in depth.

Rate limiting: Enforces a maximum number of requests in a time window. When exceeded, requests are rejected (429 Too Many Requests). Protects the service from being overwhelmed. Throttling: Controls the rate at which a service processes work — slows down processing when under stress rather than rejecting outright. Degrades gracefully instead of hard-rejecting. Throttling techniques: • Token bucket / leaky bucket: Processing proceeds at a controlled rate. Excess requests are queued (up to a limit) rather than immediately rejected. • Adaptive throttling: Dynamically adjust processing rate based on current load (CPU, latency, error rate). Gently slow down, not hard stop. • Priority queues: Under load, process high-priority requests at full speed, throttle low-priority (analytics, batch jobs). • Backpressure: Signal upstream to slow down production. In reactive streams, consumer requests only N items at a time. Differences: • Rate limiting: Applied at the boundary (API gateway). Hard rejection. User-facing. • Throttling: Applied internally. Controlled slowdown. Often invisible to caller (just slower responses). Both are complementary: Rate limiting at the edge protects from external abuse; throttling internally manages load between services. Practical: Resilience4j RateLimiter for rate limiting; combine with bounded queues + backpressure for throttling.

CQRS (Command Query Responsibility Segregation): Separate the write model (commands) from the read model (queries). Different models optimized for their respective purpose. In microservices context: • Write side: Command service receives commands (CreateOrder), validates, applies business rules, writes to the write DB, publishes domain events. • Read side: Event consumer listens to domain events, updates a denormalized read model (projection) optimized for specific queries. Often a different DB type (SQL write → Elasticsearch read). Why CQRS in microservices: • Read and write patterns often diverge significantly — a single model serves neither well • Scale reads and writes independently (read replicas for queries, vertical scale for writes) • Different services own different aspects: Order Command Service vs Order Query Service • Enables event-driven architecture naturally Challenges: • Eventual consistency between write and read models (reads may lag writes by milliseconds to seconds) • More complex data flow to reason about • Duplication of data in different forms Simple vs full CQRS: Start with simple CQRS (separate methods/classes for commands and queries, same DB). Introduce separate stores only when the read/write pattern divergence demands it. Best paired with: Event Sourcing (events are the source of truth, projections are derived).

Testing strategy for microservices — the testing pyramid: Unit tests (most): Test individual classes/functions in isolation. No external dependencies. Fast. High coverage of business logic. Use mocks for dependencies. Integration tests: Test a service with its real dependencies (DB, cache). Testcontainers spins up real Docker containers for DB/Kafka. Verifies the service works correctly with real infrastructure. Component tests: Test a complete service in isolation from other services. Use WireMock to stub external services. Verify the service's own behavior end-to-end. Contract tests (Pact): Verify that services honor the contracts they share. Consumer defines what it expects from provider. Provider verifies it can satisfy all consumer contracts. Runs in CI — catches breaking API changes before deployment. End-to-end tests (fewest): Deploy all services together and test full user journeys. Expensive, slow, brittle. Limit to critical happy paths. Service virtualization: WireMock, Hoverfly — mock entire external services for testing. Chaos testing: Inject failures (kill pods, add network latency) to verify resilience patterns work under real failure conditions. Testcontainers example: ```java @Testcontainers class OrderServiceTest { @Container PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:15"); @DynamicPropertySource static void props(DynamicPropertyRegistry r) { r.add("spring.datasource.url", postgres::getJdbcUrl); } } ```

Service isolation: The principle that each microservice is independent — its own codebase, database, deployment pipeline, and runtime. Changes to one service don't require changes to or restarts of other services. Dimensions of isolation: Data isolation: Each service has its own database. No direct DB access from other services — communicate only via API or events. Prevents schema coupling (one service's DB change breaking another service's queries). Deployment isolation: Services deploy independently. No "big bang" releases. One service releasing a bug doesn't block other services from deploying. Failure isolation: Service A's crash or memory leak doesn't take down Service B. Containers and Kubernetes namespaces enforce resource limits. Circuit breakers prevent failure propagation. Scaling isolation: Scale Service A (high CPU) independently from Service B (low load). Don't over-provision the whole monolith to handle one service's load. Team isolation (Conway's Law): Teams own services end-to-end. Team A doesn't need Team B to deploy their service. Reduces inter-team dependencies and coordination overhead. Technology isolation: Service A can use Java/PostgreSQL. Service B can use Node.js/MongoDB. Each team uses the right tool for their problem. The cost: Services communicating via network instead of in-process calls — adds latency, failure modes, and complexity. Isolation has a price; don't create microservices without clear benefits.

Event-Driven Architecture (EDA): Services communicate by producing and consuming events (immutable records of things that happened) rather than direct synchronous calls. Core components: • Event: An immutable fact that something happened (OrderPlaced, PaymentProcessed, UserRegistered). Contains event type, timestamp, payload, event_id. • Event producer: Service that publishes events when something significant happens in its domain. • Event broker: Kafka, RabbitMQ, SQS — routes events to interested consumers. Provides durability and ordering. • Event consumer: Service that subscribes to events it cares about and reacts accordingly. Benefits: • Loose coupling: Producer doesn't know about consumers. New consumers subscribe without changing the producer. • Scalability: Events buffered in the broker — consumers process at their own pace. Absorbs traffic spikes. • Auditability: Event log is a complete history of everything that happened. • Resilience: Consumer can be down and catch up when it restarts (Kafka retains events). Challenges: • Eventual consistency: Consumers are asynchronous — data lags. • Event schema evolution: Changing event format must be backward compatible (consumer may run old code). • Complex debugging: Business flows are distributed across many consumers and topics. • Ordering: Events within a Kafka partition are ordered; across partitions they're not. Avro/Protobuf with Schema Registry: Enforces event schema compatibility rules.

Database per service is the ideal in microservices — each service owns its data and exposes it only via APIs. In practice, this creates several challenges. Data joining: Service A needs data owned by Service B. Options: • API composition: Service A calls Service B via HTTP/gRPC and joins in memory. Simple for small result sets. • Event-driven denormalization: Service A subscribes to Service B's events and maintains a local replica of the data it needs. Faster reads, eventual consistency. • API gateway aggregation (BFF): Gateway calls multiple services and aggregates the response. Cross-service queries (reports, analytics): Extract data from all services into a data warehouse (BigQuery, Redshift, ClickHouse) via CDC pipelines. Run analytical queries there — not against production service DBs. Shared data pitfalls: • Temptation to share a DB: Avoids the above complexity but creates tight coupling — service boundary becomes the DB schema. Two services sharing a DB cannot be deployed independently. • Reading another service's DB directly: A performance anti-pattern that creates hidden coupling. If the schema changes, both services break. Migration path: Start by separating code into microservices but sharing DB (shared DB, separate schemas). Once teams are comfortable, split schemas into separate DB instances. Polyglot persistence: Service A uses PostgreSQL (relational data), Service B uses Cassandra (time series), Service C uses Redis (session data). Each service picks the right DB for its access patterns.

API composition: Aggregate data from multiple microservices in a single API call to the client. The aggregator (BFF or a dedicated aggregation service) calls multiple services and merges the results. Why needed: Clients need data that spans multiple service boundaries. Mobile apps especially can't afford N round trips to N services (bandwidth and latency constraints). Example: Product detail page needs: product info (Product Service), price (Pricing Service), inventory (Inventory Service), reviews (Review Service), seller info (Seller Service). Without composition: 5 separate client calls. With composition: 1 API call to BFF → BFF fans out to 5 services → merges and returns. Parallel calls: BFF should call independent services in parallel (CompletableFuture.allOf() in Java, Promise.all() in Node). Sequential calls multiply latencies. Partial failures: If Review Service is down, should product page fail? Or return product without reviews? Implement graceful degradation — return what you have, with empty/default for failed services. N+1 problem: Fetching a list of 20 orders, then calling User Service for each user = 21 calls. Batch: collect all user IDs, make one call to User Service with list of IDs. GraphQL: GraphQL resolvers naturally implement API composition with field-level fetching. DataLoader batches N sub-queries into 1. Good for complex, variable data requirements across multiple services.

Graceful degradation: When a dependency fails or is unavailable, the system continues to function (possibly with reduced functionality) rather than failing completely. Principle: Define what the "core" of your service is and protect it. Non-core features can degrade. Examples: • E-commerce product page: If Recommendation Service is down, show product without "You may also like" section. Core functionality (product info, buy button) still works. • Social feed: If Like Count Service is slow, show posts without like counts rather than making the user wait. • Checkout: If Fraud Detection Service times out, allow low-risk orders through with manual review flagging. Don't block all purchases. • Autocomplete: If suggestion service is unavailable, show empty suggestions — don't block the search form. Implementation techniques: • Default fallback values: Return empty lists, zero counts, null optional fields • Cached fallback: Return stale cached data from Redis when the live call fails • Circuit breaker with fallback: Resilience4j circuit breaker calls a fallback method when the circuit is open • Feature flags: Disable non-critical features in real-time during incidents • Timeout + partial response: Return results you have by a deadline; don't wait indefinitely Design principle: Separate "must succeed" from "nice to have" at the API level. Design fallbacks explicitly — don't let them be accidental.

Event Sourcing: Instead of storing the current state of an entity, store the full sequence of events that led to that state. The current state is derived by replaying events. Traditional approach: Store current state → UPDATE accounts SET balance = 900 WHERE id = 1; Event sourcing: Store events → INSERT INTO events (type, data) VALUES ('MoneyWithdrawn', {amount: 100, account: 1}); State reconstruction: Load all events for an entity, apply them in order, derive current state. Snapshot every N events for performance (don't replay 1M events every time). Benefits: • Complete audit trail: Every state change is recorded, with who did it and when • Time travel: Query state at any point in time by replaying up to that timestamp • Debug by replay: Reproduce production bugs by replaying the event sequence • Multiple projections: Same event stream can feed multiple read models • Event-driven integration: Events are already available for downstream consumers Challenges: • Event schema evolution: Can't change old events. Use event upcasting to transform old events to new format on read. • Eventual consistency: Read models may lag event stream • Query complexity: No simple SELECT for aggregate queries — must go through projections • Snapshot management: Without snapshots, replay time grows with history Best fit: Financial transactions, order management, audit-heavy domains. Not ideal for: simple CRUD, rapidly changing entities with no audit needs.

Centralized logging: In a microservices system with 50+ service instances, logs must be aggregated in one place for searching and correlation. Structured logging: Log JSON instead of plain text. Each log entry is a structured object with consistent fields: {"timestamp":"2024-01-15T10:30:00Z","level":"INFO","service":"order-service","traceId":"abc123","spanId":"def456","userId":"u789","message":"Order created","orderId":"ord-001"} Correlation ID / Trace ID: Every request gets a unique ID at the API gateway. Propagated in headers (traceparent) to all downstream services. Every log entry includes it. Enables searching all logs for one request across all services. Log aggregation stack: • ELK stack: Logstash (collect/transform) → Elasticsearch (store/index) → Kibana (search/visualize) • EFK stack: Fluentd (collect) → Elasticsearch → Kibana. Fluentd is lighter than Logstash. • Grafana Loki: Log aggregation without full-text indexing — cheaper at scale. Queries by label (service, environment). Good for "find logs for this service" but slower for "search all logs for this string." Log shipping: Services write to stdout. Container runtime captures to a file. Filebeat or Fluentd (DaemonSet in Kubernetes) ships to the central store. Log levels: ERROR (needs immediate attention), WARN (potential problem), INFO (business events), DEBUG (development). Never log in production at DEBUG level — too noisy and expensive. Security: Never log PII (passwords, card numbers, SSNs, emails in some contexts). Use log masking libraries.

Two-Phase Commit (2PC): Coordinator sends PREPARE to all participants → all lock resources and respond "ready." Coordinator sends COMMIT → all commit. If any fails, coordinator sends ROLLBACK. Problems with 2PC in microservices: • Synchronous and blocking: All participants must be available simultaneously • Coordinator SPOF: If coordinator fails after PREPARE, participants hold locks indefinitely (blocking protocol) • Long lock duration: Resources locked across entire transaction duration → high contention • Rarely used in microservices — tight coupling and availability impact Saga pattern: Break distributed transaction into local transactions with compensating transactions. No cross-service locks: Each service commits its local transaction immediately. On failure, execute compensation in reverse (refund, cancel, release). Key differences: • 2PC: Strong consistency, ACID, blocks on coordinator failure, low availability • Saga: Eventual consistency, no distributed locks, high availability, requires compensating transactions When saga fails (partial failure): If Payment Service charges and then Inventory Service fails, the Saga must trigger a compensation: refund the payment. Compensations are business-level undos, not DB rollbacks. Saga is the dominant pattern in microservices for multi-step business processes. 2PC is occasionally used in closely coupled, same-DB multi-step scenarios (not typical in microservices).

Timeouts: Every network call must have a deadline. Without timeouts, a slow or hung upstream service will indefinitely block threads, eventually exhausting the thread pool and causing service unavailability. Types of timeouts: • Connection timeout: Max time to establish a TCP connection. Short — 1-3 seconds. • Read timeout: Max time to wait for a response after connection established. Domain-specific — 5-30 seconds. • Write timeout: Max time to write the request. • Request timeout: Total end-to-end timeout for the full operation. Setting timeouts: • API calls to other services: 5-10 seconds read timeout • DB queries: Per-query timeout (JDBC socketTimeout or @Query timeout) • Kafka produce: producer.request.timeout.ms • Redis: jedis.timeout or lettuce command timeout Timeout cascade problem: Service A has 10s timeout for B. B has 10s timeout for C. Total worst case: 10+10 = 20s for A. Set timeouts cumulatively — each downstream timeout must be less than the upstream timeout. Deadlines (gRPC context): Pass a deadline through the entire call chain. Each hop reduces the remaining budget. If budget expires, fail immediately. Timeout + retry: Retry after timeout with exponential backoff. But: if the operation already executed (just slow to respond), retrying may duplicate it. Combine with idempotency keys. Monitoring: Alert on P99 latency approaching timeout threshold — that's future failures.

Scatter-Gather: Fan out a request to multiple services (scatter), collect and aggregate all responses (gather), then return a combined result. Use cases: Product search across multiple vendor catalogs, flight search across multiple airlines, price comparison across multiple providers, querying multiple data shards. Implementation: 1. Receive a single client request 2. Scatter: Send the same (or derived) requests to N parallel services simultaneously 3. Gather: Collect responses as they arrive. Handle partial responses. 4. Aggregate: Merge, sort, deduplicate, or summarize results 5. Return combined response Java implementation: ```java List<CompletableFuture<List<Result>>> futures = providers.stream() .map(p -> CompletableFuture.supplyAsync(() -> p.search(query))) .collect(Collectors.toList()); CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])) .thenApply(v -> futures.stream() .map(CompletableFuture::join) .flatMap(List::stream) .sorted(Comparator.comparing(Result::getPrice)) .collect(Collectors.toList())); ``` Partial failure handling: If 2 of 5 providers fail, return results from the 3 that succeeded rather than failing the entire request. Timeout: Set a gather deadline — return whatever you have when the deadline hits. Some slow providers can be ignored. Caching: Cache per-provider results to avoid repeated calls for identical queries.

Microservice: An independently deployable service that owns a bounded context — a cohesive set of related functionality and data. Responsible for a meaningful domain concept (Orders, Users, Inventory). Has its own DB, API, and deployment pipeline. A well-designed microservice: Small enough to be understood by a small team. Large enough to justify the operational overhead (CI/CD pipeline, monitoring, networking, service discovery). Nanoservice (anti-pattern): An overly fine-grained service that is too small to justify its overhead. A service with just one function, one method, or one CRUD operation. Problems with nanoservices: • Chatty: A single user action may require 20+ synchronous calls across 20 nanoservices → high latency, failure amplification • Overhead: Each service needs its own pipeline, monitoring, logging, certificate, resource quota • Logic scattering: Business logic that should live together is scattered across many services • Deployment complexity: Coordinating deployments across 200 nanoservices is impractical Right-sizing: Start coarse-grained, split when: team growth creates coordination overhead, different scaling requirements, technology diversity is clearly needed, clear domain boundary emerges. Rule of thumb: A service should be "as small as possible, as large as necessary." If a service is too small to deploy independently without always changing another service alongside it, it's too small.

Canary deployment: Route a small percentage of traffic (1-10%) to the new version. Monitor metrics. If healthy, gradually increase to 100%. If issues found, instantly roll back to 100% old version. Named after coal miners using canary birds to detect toxic gas — the canary (small traffic slice) detects problems before affecting all users. Implementation approaches: API Gateway / Load Balancer traffic splitting: Route 5% to new version pods, 95% to old. AWS ALB weighted target groups, Istio VirtualService weights, NGINX upstream weights. Kubernetes + Argo Rollouts: Declarative progressive delivery. Define canary steps: ```yaml steps: - setWeight: 5 - pause: {duration: 10m} - setWeight: 20 - pause: {duration: 10m} - setWeight: 100 ``` Auto-promote if error rate < threshold; auto-rollback if exceeded. Feature flags: New code deployed everywhere, feature flag enables it for a percentage of users. LaunchDarkly, Unleash, Flipt. Key metrics to monitor during canary: • Error rate (5xx, exceptions) • P99 latency • Business metrics (conversion rate, order completion rate) • CPU and memory usage Rollback: Shift all traffic back to old version. With Argo Rollouts, one command: kubectl argo rollouts abort my-rollout.

Service Registry: A database of available service instances with their network locations (host:port). Services register themselves on startup and deregister on shutdown (or when health checks fail). Purpose: In dynamic environments where service IPs change (container restarts, auto-scaling), a registry provides a live, up-to-date map of what's running and where. Key operations: • Register: Service registers its instance on startup: my-service, 10.0.1.5:8080, healthy • Heartbeat: Service sends periodic heartbeats. If heartbeats stop, registry removes the instance. • Deregister: Service deregisters gracefully on shutdown. • Query: Clients or load balancers query the registry to find instances for a service name. Implementations: • Eureka (Netflix/Spring): Pull-based. Clients cache registry. Self-preservation mode during network partitions. • Consul: Health check-based (HTTP/TCP/script checks). Also provides KV store, distributed locking, DNS interface. • etcd: Distributed KV store used by Kubernetes for its service registry. • Kubernetes: Built-in registry. Services get DNS names. kube-dns resolves service names to ClusterIPs. Endpoints controller maintains pod IP list per service. Kubernetes replaces explicit service registries: Services register by creating a Service object and adding label selectors. Kubernetes automatically manages the endpoint list. Most teams on Kubernetes don't need Consul or Eureka.

Blue-green deployment: Maintain two identical production environments — "blue" (current live) and "green" (new version). After testing green, switch all traffic to green instantly. Blue becomes standby for immediate rollback. Steps: 1. Blue environment is live, serving all production traffic 2. Deploy new version to Green environment (dark — no live traffic) 3. Run smoke tests against Green 4. Switch load balancer / DNS to point to Green (near-instant) 5. Green is now live 6. Keep Blue running for rollback window (1-24 hours) 7. If issues: switch back to Blue immediately 8. After validation: Blue becomes the new staging for next deployment Advantages: • Zero downtime deployment • Instant rollback (just switch back) • Full testing in production-like environment before cutover • Clean cut — no mixed-version state Disadvantages: • Double infrastructure cost (two full environments running) • DB schema must be backward compatible (both versions need to work with the same DB during transition) • Stateful services are tricky — sessions in Blue won't exist in Green Kubernetes implementation: Two Deployments (blue and green). Service selector switches between them by changing label selector. Vs rolling deployment: Rolling has mixed-version state briefly. Blue-green has instant cutover — better for major version changes with incompatibilities.

Aggregator: A service or API gateway function that calls multiple microservices, aggregates the results, and returns a unified response to the client. Why needed: Client should not know about internal service boundaries. A mobile app shouldn't need to call 5 services to render one screen — each call adds latency and battery drain. Types: Simple aggregation: Fetch data from N services, combine into one response. Example: Product detail page calling Product, Price, Inventory, and Review services. Chained aggregation: Service A calls Service B, which calls Service C. The chain builds up the response. Prone to latency chaining — if any service is slow, everything waits. Branch aggregation: Service calls two different chains and combines their results. Better when the branches are independent. Implementation: • Parallel calls: Use CompletableFuture.allOf() or reactive operators (Flux.merge, Mono.zip) to call services simultaneously. • Timeout: Set an overall deadline. Return partial results if some services are slow. • Cache: Cache upstream service responses that are stable (product name doesn't change per minute). • Error handling: Decide per dependency — some are critical (product info), some optional (recommendations). GraphQL as aggregator: GraphQL schema defines the aggregated data model. Resolvers fetch from different services. DataLoader batches calls. Natural aggregation layer.

Configuration management: Providing the right configuration to each service in each environment without hardcoding values or manual file editing. Tiers of configuration: • Static config: Baked into the image (not recommended for env-specific values) • Environment variables: 12-factor app approach. Set at deployment time. Simple and universal. • Config files: application.yml / .env files mounted via Kubernetes ConfigMaps. Good for structured, multi-key config. • Config service: Centralized service that distributes config dynamically. Spring Cloud Config Server: Git-backed config server. Services fetch their config from the server at startup (or on refresh). Config changes in Git → push to Config Server → services refresh. Supports encryption of sensitive values. Kubernetes ConfigMaps and Secrets: • ConfigMap: non-sensitive config (timeouts, feature flags, API endpoints) • Secret: sensitive values (DB passwords, API keys) • Mount as environment variables or volume-mounted files Environment-specific config: Same service code, different config per env (dev/staging/prod). Managed via environment-specific config files, Helm values files, or Kustomize overlays. Dynamic config / feature flags: Runtime changes without restart. Redis-backed feature flags, LaunchDarkly, Unleash. Service polls for flag changes or receives push notification. Best practice: Never commit secrets to Git. Use sealed-secrets or external-secrets-operator for Kubernetes. Rotate credentials regularly.

Proxy pattern: Introduce a proxy service that sits between services and acts as an intermediary, adding behavior (security, logging, transformation) without modifying the original service. Types: Service proxy: An intermediary between the client and the service. Can add: • Authentication/authorization • Rate limiting • Logging and auditing • Request transformation (add headers, modify payload) • Response transformation (filter/enrich response) • Caching Forward proxy: Client-side proxy. Client sends all outbound requests through it. Used to control and monitor outbound calls, enforce policies. Reverse proxy: Server-side proxy. All incoming requests go through it before reaching the service. Nginx, HAProxy, Envoy. Sidecar proxy: Deployed alongside each service container. Intercepts all traffic in both directions. Foundation of service mesh (Envoy in Istio). API Gateway as proxy: External clients → API Gateway (authenticates, rate-limits, routes) → backend services. Use cases in microservices: • Intercepting calls to add tracing headers without modifying the service • Rate limiting outbound calls to external APIs (per-service budget) • Circuit breaking at the proxy layer • Protocol translation (HTTP/1.1 → HTTP/2) • TLS termination/origination Envoy Proxy: The de facto sidecar proxy in modern microservices. Extremely configurable via xDS API. Core of Istio, AWS App Mesh, Consul Connect.

Distributed lock: Ensures only one service instance (across many pods/servers) executes a critical section at a time. Needed when horizontal scaling introduces race conditions. Use cases: Scheduled job should run on exactly one node, inventory decrement, leader election, cache warming (only one instance should warm on startup). Redis-based locking (Redisson / SETNX): ```java RLock lock = redissonClient.getLock("invoice-job-lock"); try { if (lock.tryLock(0, 30, TimeUnit.SECONDS)) { runInvoiceJob(); } } finally { lock.unlock(); } ``` SET key value NX PX 30000: NX = only set if not exists. PX 30000 = expire after 30 seconds (auto-release if holder crashes). Redlock (Redis multi-node): Acquire lock on majority of N Redis nodes. More fault-tolerant. Controversial — Martin Kleppmann's critique is worth reading. Zookeeper / etcd: Consensus-based distributed locking. More reliable than Redis for critical locks — single Zookeeper node failure doesn't lose the lock. Higher latency (~ms vs µs for Redis). Database advisory locks: PostgreSQL pg_advisory_lock(key) — lock at DB level. Reliable if using one DB. MySQL GET_LOCK(name, timeout). Kubernetes leader election: Kubernetes Lease object — pods compete to hold a Lease. Controller manager, scheduler use this natively. Important: Always set a lock TTL (prevents deadlock if holder crashes). Verify you still hold lock before committing critical section results.

Apache Kafka: A distributed, partitioned, replicated commit log. Often called an "event streaming platform" or "distributed log." Core characteristics: • Persistent log: Messages are written to disk and retained for a configurable period (days/weeks). Not deleted after consumption. • Partitioned: Each topic is split into partitions. Enables parallel consumption. Messages within a partition are ordered. • Replicated: Each partition has N replicas across brokers. Leader handles reads/writes; followers replicate for HA. • Consumer groups: Multiple consumers in a group each read from a subset of partitions. Horizontal scale-out. Independent consumer groups each read all messages independently. Why popular in microservices: • Decoupling: Producers and consumers are completely independent. Producer doesn't know about consumers. • Replay: Consumer crashed and missed messages? Seek back to missed offset and reprocess. Impossible with traditional queues. • Event log: Kafka as the system of record for events — event sourcing, audit log. • High throughput: Millions of messages per second per cluster. • Fan-out: N consumer groups all get all events — no single consumer bottleneck. • Stream processing: Kafka Streams, Flink, Spark Streaming integrate natively. Key concepts: Topic, Partition, Offset (consumer's position), Consumer Group, Producer, Broker, ZooKeeper/KRaft (coordination).

Service dependencies create coupling — changes in one service may break another. Managing them is critical for independent deployability. Types of coupling: • Runtime dependency: Service A calls Service B synchronously at request time. If B is down, A fails. Mitigate with circuit breakers, async messaging, caching. • Schema dependency: Service A reads Service B's DB directly. If B changes schema, A breaks. Eliminate: access data only via API. • API contract dependency: Service A depends on Service B's API shape. Mitigate with versioning and consumer-driven contracts. • Temporal dependency: Service A must be deployed before Service B. Avoid: B should work with both old and new A behavior. Reducing coupling: • Async messaging over sync calls where possible • API versioning with backward compatibility • Anti-corruption layer: Service A translates B's model to its own domain model — insulates A from B's changes • Shared nothing architecture: Minimize shared state Consumer-driven contracts (Pact): Consumer defines what it needs from provider. Provider runs contract tests in CI. Breaking the contract fails the provider's CI before deployment. Catches breaking changes automatically. Dependency map: Document service dependencies (a service dependency graph). Identify critical paths, services with many dependents, circular dependencies. Circular dependencies = design problem — extract common functionality into a shared service or event.

Bounded context: A Domain-Driven Design (DDD) concept — a defined boundary within which a domain model is consistent and specific. Within the boundary, terms have a single, unambiguous meaning (ubiquitous language). Example: "Customer" in Sales context means a prospect who has purchased. "Customer" in Support context means anyone with a ticket. "Customer" in Billing context means the account being charged. Different attributes, different behavior — despite using the same word. Forcing one unified Customer entity across all contexts leads to a bloated, confused model that tries to serve all contexts poorly. Bounded context → Microservice boundary: Each microservice should correspond to one bounded context. Services don't share domain models — they translate at the boundary (anti-corruption layer or API translation). Context map: A diagram showing how bounded contexts relate. Integration patterns: • Shared kernel: Two contexts share a subset of the domain model — coordinate tightly • Customer/Supplier: One context feeds the other — upstream/downstream relationship • Conformist: Downstream conforms to upstream's model • Anti-corruption layer: Downstream translates upstream's model to its own • Open Host Service: Well-defined public API for integration Identifying boundaries: Look for natural language shifts, different teams owning different parts, places where "it depends on context" ambiguity exists.

Request-reply over messaging: Enable synchronous-style request-response semantics over an asynchronous messaging system (Kafka, RabbitMQ). Why: Sometimes you need a response from a service but want the decoupling benefits of async messaging. Or the responding service is event-driven and doesn't expose an HTTP endpoint. RabbitMQ implementation: 1. Requester creates a temporary reply queue (or uses exclusive auto-delete queue) 2. Sends message with replyTo: my-reply-queue and correlationId: uuid 3. Responder processes message, publishes reply to replyTo queue with same correlationId 4. Requester receives message from reply queue, matches by correlationId 5. Return result to caller Kafka implementation (Spring Kafka ReplyingKafkaTemplate): 1. Send request to orders-request topic with replyTopic header and correlationId header 2. Consumer processes and produces reply to orders-reply topic with same correlationId 3. ReplyingKafkaTemplate polls reply topic, matches by correlationId, returns to caller Timeout: Set a reply timeout. If no reply within N seconds, throw an exception or return a timeout error. Temporal decoupling: Responder can reply asynchronously — doesn't need to respond before the next request arrives. Scaling concern: With Kafka, all reply-topic partitions for a consumer group receive responses — requester must read from all partitions or use a dedicated reply partition per instance.

Token Bucket: A rate limiting algorithm that controls the rate of requests while allowing bursts. How it works: • A bucket holds up to B tokens (capacity) • Tokens are added at rate R tokens per second (replenishment) • Each request consumes one token • If tokens available: consume one, allow request • If bucket empty: reject request (429) or wait Key properties: • Allows bursts: If no requests for a while, bucket fills up. Next burst of requests can use accumulated tokens. • Average rate: Over time, sustained rate cannot exceed R requests/second • Burst size: Up to B requests can be served instantly if bucket is full Vs Leaky Bucket: • Token bucket: Allows bursts (bucket fills when idle), smooth average rate • Leaky bucket: Fixed output rate regardless — no bursts, smooths input to constant rate Redis implementation: ```lua -- Lua script (atomic) local key = KEYS[1] local capacity = tonumber(ARGV[1]) local refill_rate = tonumber(ARGV[2]) local now = tonumber(ARGV[3]) local tokens = redis.call('get', key) -- compute new token count based on elapsed time + capacity ... ``` Or use Redis INCR with sliding window counter as an approximation. In practice: Bucket capacity = max burst size. Replenishment rate = sustained allowed rate. Example: 10 requests/sec average, burst up to 50: capacity=50, rate=10/sec.

Event schema evolution: Consumers may run older code when producers deploy new event formats. Must maintain backward and forward compatibility. Types of compatibility: • Backward compatible: New schema can read old data. New consumers can read old events. (Adding optional fields, not removing required fields) • Forward compatible: Old schema can read new data. Old consumers can read new events. (Removing a field is forward compatible — old consumer ignores unknown new fields it doesn't have) • Full compatible: Both backward and forward. Most restrictive. Avro with Schema Registry (Confluent): • Each schema has a version number • Compatibility rules enforced on schema registration • Consumer deserializes with the writer's schema, reads with reader's schema — automatic field mapping • Default values for new fields enable backward compatibility Protobuf evolution rules: • Never change a field's tag number • Never change a field's type • Adding new fields is safe (old consumers ignore unknown fields) • Removing fields: mark as reserved to prevent tag reuse Event upcasting: Deserialize old event format → transform to new format → process. Applied in the consumer or event store. Keeps event log immutable while allowing processing-side evolution. Versioned event types: Include version in event type name: OrderPlacedV1, OrderPlacedV2. Consumers subscribe to specific versions. Clear but creates proliferation of event types.

Monolith: Single deployable unit. All code compiled and deployed together. No enforced internal module boundaries — any code can call any other code. Simple to start, but grows into a "big ball of mud" over time — high coupling, slow build times, painful deployments. Microservices: Many independently deployable services, each with its own DB and pipeline. Strong external API boundaries enforced by the network. Maximum autonomy, maximum operational complexity. Modular monolith (recommended starting point): Single deployable unit but with enforced internal module boundaries. Modules communicate only through defined interfaces — no internal cross-module direct class calls. Each module could theoretically be extracted to a microservice when needed. Modular monolith advantages: • Simpler than microservices: No distributed system complexity (no network calls, no eventual consistency between modules) • Fast local development: No Docker-compose with 10 services • Easy refactoring: Move code between modules with IDE support — no API versioning needed • Team autonomy: Teams own modules, not files within a shared codebase • Testable: Integration tests are simple — in-process, real DB Path: Start → Modular Monolith → extract Microservices when team, traffic, or technology demands it. Examples of modular monolith frameworks: Vertical Slice Architecture, Onion/Clean Architecture, Java modules (JPMS), package-by-feature.

Choreography: Services coordinate a business process by reacting to each other's events, without a central orchestrator. Each service knows what to do when it receives a particular event. Example — order fulfillment choreography: 1. OrderService creates order → publishes OrderCreated 2. PaymentService subscribes → charges payment → publishes PaymentSucceeded 3. InventoryService subscribes to PaymentSucceeded → reserves stock → publishes StockReserved 4. ShippingService subscribes to StockReserved → schedules shipment → publishes ShipmentScheduled 5. NotificationService subscribes to ShipmentScheduled → sends email Each service is autonomous — it only knows about its own logic and the events it cares about. Advantages: • Highly decoupled: OrderService doesn't know about PaymentService • Easy to add new participants: Subscribe new service to existing events without touching other services • No central SPOF: No orchestrator that can fail Disadvantages: • Difficult to track overall flow: Business process is implicit, spread across many services • Hard to debug: Trace events across Kafka topics and multiple services • Hard to handle complex branching and error recovery • Risk of cyclic events (A triggers B triggers A) Complexity signals: If you find yourself creating a "Process Manager" service that subscribes to events and coordinates — you've accidentally built an orchestrator. Consider switching to explicit orchestration.

Composite microservice: A service that implements its functionality by composing calls to other primitive/atomic microservices. Acts as an orchestrator for a specific business operation. Also called: Process service, Orchestration service, Aggregation service. Example: • Product Search Service (composite): Calls Product Service + Inventory Service + Pricing Service + Review Service in parallel, merges results, applies business logic (sort, filter, rank), and returns a unified search result. • Order Processing Service (composite): Orchestrates the order creation saga — calls Payment Service, Inventory Service, and Shipping Service in sequence, handling failures with compensation. Why composite services: • Complex business operations span multiple atomic services • Client shouldn't be responsible for orchestration • Centralized error handling, compensation logic, and retry policies for a specific flow Composite vs BFF: BFF is client-specific aggregation at the edge. Composite service is internal, reusable by multiple clients or other services. Design principle: Atomic services contain domain logic and data. Composite services contain workflow logic. Separation prevents atomic services from becoming tightly coupled. Tradeoff: Composite services introduce a fan-out dependency — if it calls 5 services, all 5 must be healthy for the composite to succeed. Mitigate with circuit breakers, fallbacks, and optional dependencies.

Chatty microservices: A single user operation requires many synchronous API calls between services, causing high latency, tight coupling, and amplified failure risk. Example: Loading a dashboard makes 15 sequential API calls across 8 services → 15 network round trips × average 20ms each = 300ms minimum, before any business logic. Solutions: 1. Aggregate with BFF: Build a Backend For Frontend that makes all 8 calls in parallel and returns one combined response. Client makes 1 call, BFF handles the fan-out. 2. Event-driven denormalization: Instead of calling User Service for username on every request, subscribe to User events and maintain a local copy of the data you need. Read locally — zero network calls. 3. API composition at gateway: GraphQL BFF resolves fields from multiple services in a single request. Client specifies exactly what fields it needs. 4. Service redesign: If services A and B always need to call each other, they may be the wrong services. Consider merging into one service or redesigning the boundary. 5. Coarser-grained APIs: Instead of fine-grained CRUD endpoints, provide higher-level business operation endpoints that do more in one call: /checkout (not separate calls to /cart, /inventory, /payment). 6. Async where possible: Not all chatty calls need to be synchronous. Background sync for non-critical data reduces runtime call chains. 7. Client-side caching: Cache frequently-read, infrequently-changed data (user profile, product metadata) in the caller service.

Process manager (workflow engine): Coordinates long-running, multi-step business processes across multiple services. Maintains the state of where the process is, what has been done, and what to do next. Difference from simple Saga: A Saga handles one business transaction with compensations. A process manager handles longer-lived workflows with complex branching, waiting for external events (human approval, third-party callback), timeouts, and retries. Why needed: Business processes can span hours/days (order → fulfillment → shipping → delivery → review request). The process manager survives service restarts and tracks state durably. Example workflow: 1. Order placed → start workflow 2. Check fraud score (wait for ML model) 3. If fraud score high → manual review queue → wait for human decision 4. Payment charged 5. Inventory reserved 6. Shipping scheduled 7. Wait 3 days for delivery 8. Send review request email Tools: • Temporal: Code-based workflow orchestration. Workflows are regular code — Java/Go/Python. Durable execution — automatically replays on failure. Handles long waits (years). Used by Stripe, Coinbase, Snap. • Netflix Conductor: JSON-defined workflows, REST API to start/query. • Apache Airflow: DAG-based workflow orchestration (more for data pipelines than business processes). • Camunda: BPMN-based (business process model notation). Good for business-visible workflows.

Zero-downtime deployment: Deploy new versions without any service interruption for users. Kubernetes rolling update (default): • Replaces old pods with new pods one at a time (or in batches) • New pods are health-checked before old ones are removed • At any point, both versions are running — API must be backward compatible • Rollback: kubectl rollout undo deployment/my-service Graceful shutdown: • Service must handle SIGTERM: stop accepting new connections, finish in-flight requests, then exit • Spring Boot: server.shutdown=graceful + spring.lifecycle.timeout-per-shutdown-phase=30s • Kubernetes: preStop hook + terminationGracePeriodSeconds to allow drain DB schema compatibility: • New code must work with old schema (during rollout, old pods still running) • Old code must work with new schema (during rollback) • Use expand-contract: add nullable columns first, deploy code, backfill, add constraints • Never remove a column in the same deployment that stops writing to it Session handling: • Stateless services: No issue — any pod handles any request • Stateful sessions: Store in Redis — new pods can access sessions created by old pods Message consumers: • Kafka: New consumer group version picks up where old left off (same group ID) • Ensure consumer is idempotent — may reprocess messages during restart

Anti-Corruption Layer (ACL): A translation layer between two bounded contexts or systems that prevents the model of one system from "corrupting" (bleeding into) the other system's domain model. Why needed: When Service A integrates with Service B (legacy system, third-party API, or different bounded context), B's concepts and terminology may be incompatible with A's domain model. Without an ACL, A starts using B's terminology and logic throughout its code — corrupting its own clean domain model. Example: Your domain has "Customer" with id, name, preferences. The legacy CRM has "Account" with account_number, full_name, settings_json. Without ACL: Customer code becomes polluted with CRM concepts. With ACL: ACL translates CRM Account → your Customer model. Rest of your code never knows the CRM exists. Implementation: • Adapter: Translates between the external system's API and your domain model • Facade: Simplifies complex external interface into a clean internal API • Translator: Maps external data types, field names, enums to your domain equivalents Where to use: • Integrating with legacy systems • Consuming third-party APIs (payment gateways, shipping providers) • Two microservices with genuinely different domain models • Migrating from monolith to microservices (new service insulates from monolith's model) Benefit: You can replace or modify the external system without changing your domain code — just update the ACL translation.

Common microservices pitfalls: 1. Premature adoption: Building microservices before understanding the domain. Extract services only when you have clear boundaries and actual scaling/team needs. 2. Nano-services: Too-small services with no cohesion. One service per DB table, one service per operation. Leads to chatty, tightly coupled systems. 3. Distributed monolith: Services that are deployed independently but share the same DB or call each other in tight, synchronous chains. Gets the worst of both worlds. 4. Synchronous everything: Wiring all services together with synchronous HTTP calls. Any failure cascades immediately. Should use async messaging for decoupled operations. 5. Ignoring data ownership: Allowing services to access each other's DBs directly. Creates hidden coupling — schema changes break multiple services. 6. No observability: Not setting up distributed tracing, centralized logging, and metrics from day one. Debugging production issues across 20 services without tracing is nearly impossible. 7. No API versioning strategy: Services deployed independently break each other when APIs change. Need versioning and consumer-driven contracts. 8. Organizational mismatch: Splitting services along technical layers (frontend service, DB service) instead of business domains. Conway's Law — services should match team boundaries. 9. Too much coordination needed: If deploying Service A always requires coordinating with 3 other teams — the boundaries are wrong. 10. Ignoring operational complexity: Microservices multiply operational surface area. Need service discovery, config management, secrets, health checks, distributed tracing, from day one.

Both coordinate multi-step workflows across microservices — they differ in where the control logic lives. Orchestration: Central coordinator (orchestrator) explicitly tells each participant what to do and in what order. Orchestrator is the brains — it knows the entire workflow. Analogy: A conductor directing an orchestra. Each musician follows the conductor's signals. Characteristics: • Single source of truth for workflow state • Easier to understand the full process (all logic in one place) • Easier to monitor (query orchestrator for workflow status) • Easier to handle complex branching, error recovery, retries • Orchestrator becomes a central dependency Choreography: Services react to events and emit their own events. No central coordinator — each service knows its role when specific events occur. Analogy: Jazz improvisation — each musician reacts to others without a conductor. Characteristics: • Highly decoupled — services don't know about each other • Easy to add new participants (subscribe to existing events) • Business flow implicit — must trace events to understand the process • Harder to debug and monitor • Risk of event cycles Practical guidance: Use orchestration for complex, business-critical workflows with compensations and branching (Saga orchestration with Temporal). Use choreography for simple event fan-out where many consumers react to one event independently.

Consistent read: Ensuring a client sees a logically consistent view of data that spans multiple services — not a mix of old and new state from a partially completed operation. Challenge: Service A writes data. It publishes an event. Service B consumes and updates its read model. Client queries Service A (gets new data) and Service B (may still have old data) in the same request — sees inconsistency. Approach 1 — Accept eventual consistency: For most use cases, slight inconsistency is acceptable. Design the UI to handle it (show "updating..." states, optimistic UI updates). The system converges — just not instantly. Approach 2 — Read-your-writes from same service: After a write to Service A, only read from Service A for the data that was just written. Route reads to the primary after writes (for DB read replicas within a service). Approach 3 — Synchronous calls: For data requiring immediate consistency, have Service A synchronously call Service B and update it during the write transaction (tight coupling — use sparingly). Approach 4 — Versioned consistency tokens: Service A returns a version/timestamp after write. Client includes this token in subsequent reads. Services check they have data at least as current as the token before responding. Approach 5 — Single source of truth: Query Service A only. Service B's data is a materialized view for a specific read use case — don't mix sources in the same logical operation. Design advice: Minimize cross-service reads for operations requiring consistency. Design service boundaries so consistent data lives in one service.

Load shedding: Deliberately dropping or rejecting requests when a service is at capacity, to protect itself from being overwhelmed and to maintain acceptable latency for requests it does handle. Philosophy: Better to explicitly reject 20% of requests with a fast error than to accept all requests and serve 100% of them slowly (or cause the service to crash, serving 0%). Techniques: 1. Request queue depth limit: Reject new requests when the internal processing queue exceeds a threshold. Fast rejection before the queue grows to cause memory issues or massive latency. 2. Latency-based shedding: If P99 latency exceeds a threshold, start shedding low-priority requests to protect high-priority ones. 3. CPU-based: If CPU usage exceeds 80%, reject excess requests. Prevents CPU saturation from causing response time degradation for all requests. 4. Priority-based: Classify requests into tiers. Under load, first shed batch/background jobs, then non-critical user features, then critical user actions, last. 5. Token bucket at service entry: Per-service internal rate limit — excess requests get 503, not queued indefinitely. Response: Return 503 Service Unavailable with Retry-After header. Clients should respect this and back off. Kubernetes: Resource limits + HPA (Horizontal Pod Autoscaler) auto-scale to meet demand instead of shedding — but scaling has limits and lag. Load shedding is the last resort within the scaled capacity. Go analogy: A busy restaurant seats customers at available tables. When full, new arrivals wait or are told the wait time. Load shedding is telling the 51st customer when the restaurant seats 50: "Come back in 30 minutes."

REST: • Protocol: HTTP/1.1 (or HTTP/2) • Data format: JSON (human-readable, self-describing) • Contract: OpenAPI/Swagger (optional, not enforced) • Type safety: None by default — no compile-time check that response matches expected shape • Tooling: Universal — any HTTP client, browser, curl • Streaming: Limited (SSE, chunked transfer) gRPC: • Protocol: HTTP/2 (binary framing, multiplexing, header compression) • Data format: Protocol Buffers (binary, compact, typed) • Contract: .proto files — strongly typed, auto-generates client+server stubs in 12+ languages • Type safety: Compile-time checked via generated code • Tooling: Needs gRPC client — can't use curl directly (use grpcurl or grpcui) • Streaming: First-class (unary, server-stream, client-stream, bidirectional) Performance: gRPC payload is ~5-10× smaller than JSON. Connection multiplexed over HTTP/2. Benchmark shows gRPC 4-10× faster than REST for equivalent data. When to use REST: • External/public APIs (browser clients, third-party consumers) • Simple CRUD services where tooling simplicity matters • Teams unfamiliar with protobuf workflow When to use gRPC: • Internal service-to-service communication • Low-latency requirements • Streaming use cases (real-time data, bidirectional chat) • Polyglot microservices (generate clients in Java, Python, Go from one proto file)

Both are API Gateway responsibilities — but serve different purposes. Gateway Routing: Simple proxy. Gateway receives request → looks at URL/path → routes to the appropriate backend service → returns response unchanged. Example: GET /api/orders → Order Service, GET /api/users → User Service, POST /api/payments → Payment Service. No data transformation. No aggregation. The gateway is a dumb router. Gateway Aggregation: Gateway calls multiple backend services, aggregates the responses, and returns a combined result to the client in a single call. Example: GET /api/dashboard → Gateway calls User Service + Order Service + Notification Service in parallel → combines { user: {...}, recentOrders: [...], notifications: [...] } → returns to client. Benefits of aggregation: • Reduces client-side round trips (3 calls → 1 call) • Client is simpler — doesn't need to know about internal services • Gateway optimizes parallel calls internally BFF (Backend For Frontend) is the full realization of gateway aggregation — a dedicated aggregation layer per client type. When gateway aggregation is appropriate: • Common data combinations needed by many clients • Mobile clients where reducing round trips is critical • Stable aggregation patterns that don't change frequently Avoid: Putting complex business logic in the gateway. The gateway should aggregate, not compute. Keep business logic in the services.

Inbox pattern: Guarantees exactly-once processing of incoming messages by recording received messages in an inbox table before processing. The counterpart to the Outbox pattern. Problem: Message brokers guarantee at-least-once delivery. A consumer may receive the same message multiple times (retry after timeout, broker failure). Without deduplication, business logic executes multiple times — double charge, double order. Solution: 1. Receive message from broker 2. Insert into inbox table: (message_id, payload, received_at, status='pending') 3. Use message_id as unique constraint — duplicate message → constraint violation → ignore 4. Process the pending message, update status to 'processed' Atomic processing: Process message AND update inbox status in the same DB transaction. If processing fails, both roll back — message stays pending and will be retried. Inbox table: { message_id VARCHAR (PK), payload JSON, received_at DATETIME, processed_at DATETIME, status } Cleanup: Archive or delete processed inbox entries after retention period. Vs idempotency key: Inbox is a comprehensive deduplication mechanism. Idempotency key is a per-operation check. Inbox at message level, idempotency at business operation level — both are valid and complementary. When to use: When the downstream operation is not naturally idempotent and exactly-once semantics are critical (payment processing, order creation).

Event versioning is critical — consumers may be running older code when producers deploy new event formats. Approach 1 — Schema Registry with Avro/Protobuf: Produce: Serialize event with schema → schema ID embedded in message header. Consume: Schema ID extracted → fetch schema from registry → deserialize. Registry enforces compatibility rules (BACKWARD, FORWARD, FULL). Prevents publishing incompatible schemas. Approach 2 — Add version field to event: {"version": 2, "type": "OrderPlaced", "payload": {...}}. Consumers check version field and apply appropriate deserialization/mapping logic. Simple but requires branching in consumer code. Approach 3 — New topic per major version: When a breaking change is necessary, publish to a new topic: orders-v1 → orders-v2. Consumers migrate to new topic at their own pace. No forced simultaneous migration. Old topic deprecated and eventually removed. Compatibility rules for schema evolution: • Adding optional fields with defaults: BACKWARD compatible (old consumer ignores new field) • Removing optional field: FORWARD compatible (old events still valid for new consumer) • Adding required field without default: NOT compatible — don't do this • Renaming field: NOT compatible — add new field, deprecate old field, remove later Consumer resilience: Consumers should handle unknown fields gracefully (ignore them) — enables forward compatibility. Monitoring: Track consumer lag per schema version. Alert when old consumers haven't migrated past a deadline.

Open Host Service (OHS): A bounded context exposes a well-defined, published protocol (API) for others to integrate with. The protocol is explicitly designed for external consumption — stable, versioned, and documented. Context: In DDD integration patterns, this is when an upstream context deliberately makes integration easy for downstream contexts. The upstream "opens" its service as a first-class integration surface. Characteristics: • Defined explicitly for integration, not just incidentally exposed • Versioned: Changes are backward compatible or add a new version • Published: Documented (OpenAPI, GraphQL schema, gRPC proto files) • Stable: Breaking changes are rare and follow a deprecation process • Serves multiple consumers without requiring them to adapt to internal domain details Example: A User service exposes an OHS: • POST /api/v1/users (create user) • GET /api/v1/users/{id} (get user) • GET /api/v1/users?email=... (find by email) • Published protocol: users.proto or openapi.yaml Other services integrate via this clean API, not by reading the users DB directly. Vs generic subdomain: OHS is the pattern for sharing reusable domain capabilities. If the User Service provides user management as a service to the entire company, it's both a generic subdomain and an OHS. Related: Consumer teams may use an Anti-Corruption Layer (ACL) to translate the OHS protocol into their own internal domain model — keeping their model clean even when consuming the OHS.

SLO (Service Level Objective): An internal target for a service quality metric. Defines what "healthy" means for your service. Per-service SLOs: • Availability: 99.9% of requests return non-5xx response • Latency: P99 response time < 200ms • Error rate: < 0.1% of requests result in errors • Throughput: Handle 1000 RPS sustained without degradation Implementing measurement: • Instrument with Micrometer (Spring Boot): HTTP server metrics auto-collected (request count, response time histogram, error count) • Export to Prometheus: Scrape at 15s intervals • Alert rules: P99 latency > 200ms for 5 minutes → alert Error budget: SLO = 99.9% → error budget = 0.1% per month = 43.8 minutes downtime/month. When budget is consumed, halt feature work and focus on reliability. Multi-service SLO: For a user-facing operation that spans 5 services, the end-to-end SLO is the product of component SLOs: 99.9%^5 = 99.5%. Composition erodes reliability — keep dependencies minimal and async where possible. SLO tracking in Grafana: Time-series panel showing error rate and latency vs thresholds. Burndown chart showing error budget consumption over the month. Review cadence: Review SLO adherence weekly in team meetings. Major violations trigger post-mortems. Quarterly review: are SLOs too strict (wasted engineering), too loose (users unhappy)?

Claim Check (also called Reference-Based Messaging): When a message payload is too large for the message broker, store the large payload in external storage and send only a reference (claim check ticket) in the message. Consumer uses the reference to retrieve the full payload. Why needed: Message brokers have message size limits (Kafka default max is 1MB, SQS is 256KB). Large payloads cause performance issues — serialization overhead, network bandwidth, broker memory. Pattern: 1. Producer checks payload size 2. If large (> threshold): Upload payload to S3 (or blob storage). Store the S3 URL/key in the message: {"type": "ReportGenerated", "s3Key": "reports/2024/annual-report.json"} 3. Send lightweight message to broker with the reference 4. Consumer receives message, retrieves payload from S3 using the key 5. Process the full data Trade-offs: • Additional storage cost (S3) • Extra network round trip for consumer to fetch payload • Lifecycle management: When to delete from S3? TTL or explicit cleanup. Alternatives: • Compress the payload before messaging (gzip JSON) • Redesign message to include only changed fields (delta), not full snapshot • Break large message into multiple smaller events When to use: Report generation results, batch data imports, large document processing, ML model outputs.

Service mesh traffic management: Controlling how traffic flows between services — routing, load balancing, retries, circuit breaking — at the infrastructure layer without application code changes. Key capabilities (Istio VirtualService example): Weighted routing (canary): ```yaml http: - route: - destination: {host: reviews, subset: v3} weight: 10 - destination: {host: reviews, subset: v2} weight: 90 ``` Fault injection (chaos testing): ```yaml fault: delay: percentage: {value: 50} fixedDelay: 5s abort: percentage: {value: 10} httpStatus: 503 ``` Retry policy: ```yaml retries: attempts: 3 perTryTimeout: 2s retryOn: gateway-error,connect-failure ``` Timeout: ```yaml timeout: 5s ``` Circuit breaking (DestinationRule): ```yaml outlierDetection: consecutiveErrors: 5 interval: 30s baseEjectionTime: 30s ``` Benefits: Consistent policies across all services without per-service code. A/B testing by routing specific users to specific versions. Production chaos testing via fault injection. Traffic mirroring — shadow copy of production traffic to new version for testing.

Vertical scaling (scale up): Add more CPU, RAM, or faster disk to existing instances. Simple — no architectural change. Limited by hardware ceiling. Single point of failure remains. Useful for: stateful services (DBs) where horizontal scaling is complex. Horizontal scaling (scale out): Add more instances. Requires stateless services (any instance can handle any request). Load balancer distributes traffic. Unlimited theoretical scale. Fault-tolerant (instance failure affects only a fraction of capacity). Microservices + horizontal scaling: • Each service scales independently: Order Service at 10 pods, Payment Service at 2 pods, Report Service at 1 pod — right-size each service • Kubernetes HPA (Horizontal Pod Autoscaler): Scales based on CPU, memory, or custom metrics (Kafka lag, RPS) • Kubernetes VPA (Vertical Pod Autoscaler): Adjusts resource requests/limits based on actual usage Stateful challenges with horizontal scaling: • Session state: Use Redis or JWT (stateless token) — not in-memory • In-memory caches: Each pod has its own cache → stale reads → use Redis cluster instead • Scheduled jobs: Only one instance should run → distributed lock or Kubernetes CronJob with concurrencyPolicy: Forbid • Sticky connections (gRPC): Client connects to one server for streaming — load balancer must support sticky sessions or connection-level routing KEDA (Kubernetes Event-Driven Autoscaling): Scale based on Kafka consumer lag, SQS queue depth, or any custom metric — not just CPU.

Schema Registry: A centralized service that stores and enforces schemas for messages exchanged between services. Ensures producers and consumers agree on message format without direct coordination. How it works: 1. Producer registers schema (Avro, Protobuf, JSON Schema) in registry → gets schema ID 2. Producer serializes message with schema → embeds schema ID in message header → sends to Kafka 3. Consumer receives message → extracts schema ID → fetches schema from registry → deserializes 4. If producer tries to register an incompatible schema → registry rejects with compatibility error Compatibility types: • BACKWARD: New schema can read data written with old schema (consumers can upgrade first) • FORWARD: Old schema can read data written with new schema (producers can upgrade first) • FULL: Both backward and forward (most restrictive, safest) • NONE: No compatibility check Conflent Schema Registry: De facto standard. Supports Avro, Protobuf, JSON Schema. REST API for schema management. Kafka clients have native support. Benefits: • Prevents schema changes that would break consumers from reaching production • Schema evolution is documented and versioned • Consumers always know how to deserialize — no out-of-band documentation • Avoids "schema drift" — different services using incompatible versions of the same concept Deployment: Separate service (Confluent Registry) or embedded in Kafka (Confluent Cloud). Needs high availability — if registry is down, producers/consumers can't serialize/deserialize.

Kubernetes has built-in service discovery that replaces the need for Eureka, Consul, or Zookeeper for most microservices use cases. Kubernetes Service object: Creates a stable DNS name and virtual IP (ClusterIP) for a set of pods. kube-proxy manages routing from ClusterIP to actual pod IPs. Pods come and go — Service is stable. DNS discovery: • Within the same namespace: http://order-service:8080 • Cross-namespace: http://order-service.order-namespace.svc.cluster.local:8080 • Kubernetes DNS (CoreDNS) resolves service names to ClusterIPs Service types: • ClusterIP (default): Internal-only, reachable within cluster • NodePort: Exposed on a port on every node — for external access without load balancer • LoadBalancer: Creates cloud provider load balancer (AWS ALB, GCP LB) — external access with a stable IP • ExternalName: Maps service to an external DNS name — for integrating external services into cluster DNS • Headless (ClusterIP: None): Returns pod IPs directly — for gRPC or stateful sets needing pod-level targeting Ready check: Service only routes to pods that pass readiness probes — unhealthy pods are automatically excluded without manual deregistration. Headless services for gRPC: gRPC uses HTTP/2 multiplexing — a single TCP connection. Standard ClusterIP routes all gRPC to one pod (client-side LB needs pod IPs). Use headless service → client gets all pod IPs → client-side load balances across pods.

Pub-Sub: Publishers send messages to a topic without knowing who will receive them. Subscribers register interest in specific topics and receive all matching messages. Publisher and subscriber are fully decoupled. Key properties: • Many-to-many: Multiple publishers to one topic; multiple subscriber groups each receive all messages • Decoupled: Publisher doesn't know subscribers exist; subscriber doesn't know publisher identity • Fan-out: One event → N consumers, each processing independently Implementations: • Kafka: Topic-based pub-sub. Consumer groups each get all messages. Persistent — replay possible. • RabbitMQ fanout exchange: Publishes to all bound queues simultaneously. • AWS SNS + SQS: SNS topic fans out to multiple SQS queues. Each queue has its own consumer. SNS also supports HTTP/Lambda/email subscribers. • Redis Pub/Sub: In-memory, no persistence. Message lost if no subscriber is connected. Good for real-time notifications, not reliable messaging. • Google Cloud Pub/Sub: Managed, at-least-once delivery, push or pull model. Vs point-to-point (queue): Queue delivers each message to exactly ONE consumer (load balancing). Pub-Sub delivers each message to ALL subscriber groups (fan-out). Use cases: Broadcast events (OrderPlaced) to many interested services (Payment, Inventory, Notification, Analytics). Decouple producers from an evolving set of consumers. Event-driven microservices backbone.

An API gateway is a reverse proxy with additional cross-cutting concerns. Key capabilities to implement: 1. Routing: Map incoming URL paths/methods to backend service URLs. Path-based: /api/orders → http://order-service:8080 Header-based: X-Version: 2 → http://order-service-v2:8080 2. Authentication: Validate JWT/API key before forwarding. Extract user context from token, pass as enriched header to backend. 3. Rate limiting: Redis token bucket per (user_id or IP) per endpoint. Return 429 with Retry-After on limit exceeded. 4. Request/response transformation: Add/remove headers, rewrite paths, transform request body format. 5. Load balancing: Maintain a pool of upstream instances, health check them, distribute requests (round-robin, least connections). 6. Circuit breaking: Track upstream error rates, open circuit on threshold, fail fast, probe for recovery. 7. SSL termination: Accept HTTPS from clients, forward HTTP to backends (internal network is trusted). 8. Logging and tracing: Log all requests with trace-id, status, latency. Inject trace headers before forwarding. Using Spring Cloud Gateway: ```yaml spring.cloud.gateway.routes: - id: order-route uri: lb://order-service predicates: [Path=/api/orders/**] filters: [AuthFilter, RateLimitFilter, RequestLogging] ``` Production: Use Kong, AWS API Gateway, or Envoy rather than building from scratch. The value is in your business logic, not in re-implementing a gateway.

Consumer-driven contract testing: The consumer of an API defines a contract specifying what requests it sends and what responses it expects. The provider verifies it satisfies all consumer contracts. Catches integration breakage at build time, not at runtime. Problem with end-to-end tests: Slow, flaky, hard to maintain, require all services to be running simultaneously. Integration bugs still slip through because teams don't know which consumers depend on which response fields. Pact workflow: 1. Consumer team writes a Pact test: "When I send GET /users/1, I expect {id: 1, name: 'John', email: 'john@example.com'}" 2. Pact generates a contract file (pact JSON) 3. Contract is uploaded to Pact Broker (central server) 4. Provider (User Service) CI pulls all consumer contracts from Pact Broker 5. Provider verifies it satisfies each contract by replaying the specified request and checking the response matches 6. Verification result published to Pact Broker 7. can-i-deploy check: Consumer CI queries Pact Broker — is the provider verified for this consumer version? Only deploy if yes Benefits: • Catch breaking API changes before deployment — in CI • No shared test environment needed — consumer and provider test independently • Know exactly which consumers depend on which response fields — safe to remove unused fields • Teams move fast without fear of breaking each other Pact Broker can-i-deploy: git commit SHA + service name → verified safe to deploy to specific environment.

Feature toggles (feature flags): Enable/disable functionality at runtime without deploying new code. Decouple code deployment from feature release. Types: • Release toggles: Hide in-progress work. Remove when feature ships. Short-lived. • Experiment toggles: A/B testing. Route % of users to different code paths. Medium-lived. • Ops toggles: Kill switches for problematic features under load. Can persist long-term. • Permission toggles: Show features to specific users/roles (beta users, internal staff). Implementation: ```java if (featureFlags.isEnabled("new-checkout-flow", userId)) { return newCheckoutService.checkout(cart); } else { return legacyCheckoutService.checkout(cart); } ``` Flag evaluation: Flags evaluated per request with user context. Supports percentage rollouts (10% of users), user segment targeting, environment-specific flags. Storage: Flags stored in a config service (LaunchDarkly, Unleash, AWS AppConfig, or custom Redis-backed service). Changes take effect immediately without restart. Microservices concern: Each service evaluates flags independently. If Feature X spans 3 services, all 3 need consistent flag state — use a shared flag service or propagate flag decisions via request headers. Debt: Remove flags after feature is fully released. Flag debt accumulates — dead branches, tests for both paths, cognitive load. Enforce TTL on release toggles. Rollback: If a feature causes problems in production, disable the flag instead of rolling back deployments — instant recovery.

Polyglot persistence: Each microservice chooses the database technology best suited to its specific data model and access patterns — rather than all services sharing one central database. Why: Different services have fundamentally different data needs. Forcing all into a relational DB is a compromise that serves none of them optimally. Examples per service type: • Order Service: PostgreSQL — transactional ACID, complex joins for order history • Product Catalog: Elasticsearch — full-text search, faceted filtering, relevance scoring • User Sessions: Redis — key-value, sub-millisecond latency, TTL-based expiry • Social Graph: Neo4j — graph DB for "friends of friends", relationship traversal • Analytics / Events: Cassandra or ClickHouse — time-series writes, columnar analytics reads • Documents/Contracts: MongoDB — flexible schema, nested document storage • Recommendations: Redis with sorted sets or specialized vector DB (Pinecone) for ML embeddings Benefits: Each service uses the right tool. No one team's DB scaling decisions affect others. Teams can evolve their storage technology independently. Challenges: • Cross-service queries: No SQL JOIN across different DBs. Use API composition, CQRS read models, or data pipelines. • Operational overhead: Teams need expertise in multiple DB technologies. SRE team must monitor and operate many DB systems. • Data consistency: No distributed transactions across different DB types. Use Saga pattern. • Data replication: Analytics queries spanning multiple services require ETL pipeline or event-driven denormalization into a data warehouse.

Distributed caching: Shared in-memory data store (Redis, Memcached) accessible by multiple service instances. Reduces DB load and latency for frequently-read, infrequently-changed data. Cache-aside (lazy loading): Service checks cache first. On miss, reads from DB, writes to cache, returns result. Most common pattern. Cache only contains data that was actually requested. ```java String cached = redis.get("user:" + userId); if (cached == null) { User user = db.findById(userId); redis.setex("user:" + userId, 3600, serialize(user)); return user; } return deserialize(cached); ``` Write-through: On every DB write, also update cache synchronously. Cache is always warm — no cold misses for recently written data. Risk: cache write failure can fail the request. Write-behind (write-back): Write to cache first, async flush to DB. Fast writes, but risk data loss if cache crashes before flush. Read-through: Cache sits in front of DB. Service always reads from cache. Cache is responsible for loading from DB on miss. Simplifies application code. Cache invalidation strategies: • TTL-based: Expire after N seconds. Simple but may serve stale data. • Event-driven: When an entity is updated, publish event → subscriber deletes/updates cache entry. Consistent but complex. • Version tagging: Cache key includes version number. On update, increment version → old key naturally becomes unreferenced. Cross-service caching: Avoid caching another service's data — you won't receive invalidation events. Each service caches its own domain data. Cache stampede prevention: When cache expires, many requests hit DB simultaneously. Use probabilistic early expiration or a distributed lock on cache miss.

Microservices CI/CD: Each service has an independent pipeline. Deploy any service without coordinating with other teams. Pipelines are fast (~5-10 min) and reliable. Per-service pipeline stages: 1. Trigger: Push to main branch or PR creation 2. Build: Compile + unit tests (fast, no external deps) 3. Docker image: Build image, tag with git commit SHA 4. Integration tests: Spin up service with test DB (Docker Compose) 5. Contract tests: Run Pact consumer/provider verification 6. Vulnerability scan: Trivy or Snyk on Docker image 7. Push image: Push to container registry (ECR, GCR, Docker Hub) 8. Deploy staging: Update Kubernetes deployment (set image tag) 9. Smoke tests: Hit staging endpoints, verify health 10. Deploy production: Canary → gradual rollout → full rollout Key principles: • Immutable artifacts: Docker image built once, promoted through environments. Never rebuild — staging and prod run identical artifacts. • Gitops: Deployment state in git (ArgoCD/Flux). Actual cluster state reconciled to desired state in repo. Audit log for free. • Canary deployments: Deploy to 5% of traffic → monitor error rate and latency → if clean, promote to 100%. Automated rollback on metric degradation. • Trunk-based development: Short-lived feature branches, frequent merges to main. No long-running branches. Feature flags hide incomplete work. Monorepo vs polyrepo: • Polyrepo: One git repo per service — fully independent pipelines, but hard to make cross-service changes atomically • Monorepo: All services in one repo — shared tooling, atomic cross-service changes, but pipeline must detect what changed and only rebuild affected services (Nx, Turborepo, Bazel)

Observability stack: Micrometer (instrumentation) → Prometheus (metrics collection + storage) → Grafana (visualization + alerting). Micrometer in Spring Boot: • Add spring-boot-starter-actuator + micrometer-registry-prometheus • Exposes /actuator/prometheus endpoint with metrics in Prometheus format • Auto-instruments: HTTP request count/duration, JVM memory, GC, thread pools, DB connection pools, Kafka consumer lag Custom metrics: ```java Counter orderCounter = registry.counter("orders.created", "status", "success"); orderCounter.increment(); Timer timer = registry.timer("payment.processing.time"); timer.record(() -> paymentService.process(payment)); Gauge.builder("queue.depth", queue, Queue::size).register(registry); ``` Prometheus configuration (scrape config): ```yaml scrape_configs: - job_name: order-service static_configs: - targets: [order-service:8080] metrics_path: /actuator/prometheus scrape_interval: 15s ``` In Kubernetes: Use ServiceMonitor (Prometheus Operator) — auto-discovers all pods with specific annotations. Grafana dashboards: Pre-built dashboards for Spring Boot (JVM dashboard ID 4701), Kafka, PostgreSQL. Custom dashboards: HTTP error rate, P99 latency, business metrics (orders/min, payment success rate). Alerting: Prometheus AlertManager rules for P99 > SLO threshold, error rate spike, consumer lag growing. Routes to PagerDuty/Slack. Distributed tracing: Add Micrometer Tracing (OpenTelemetry) → traces exported to Tempo or Jaeger → Grafana Tempo for trace queries. Correlate trace-id across logs, metrics, and traces.

Retry pattern: Automatically re-attempt a failed operation, assuming transient failures (network blip, temporary overload) will resolve on their own. Exponential backoff with jitter: ```java Retry retry = Retry.backoff(3, Duration.ofSeconds(1)) .maxBackoff(Duration.ofSeconds(10)) .jitter(0.5) // randomize ±50% to prevent thundering herd .filter(ex -> ex instanceof ConnectException); // only retry specific exceptions ``` Spring Retry: ```java @Retryable(value = {HttpServerErrorException.class}, maxAttempts = 3, backoff = @Backoff(delay = 1000, multiplier = 2)) public OrderResponse callOrderService(Order order) { ... } ``` When retries help: Network timeouts, 503 Service Unavailable (transient overload), 429 Too Many Requests (rate limit — wait and retry). When NOT to retry: • 400 Bad Request: Invalid request body — retrying the same request will always fail • 401 Unauthorized / 403 Forbidden: Auth issue — retrying won't fix it • 404 Not Found: Resource doesn't exist — retrying won't create it • Non-idempotent operations: POST /charge-payment — retrying may double-charge. Only retry if the operation is idempotent or you have an idempotency key that prevents duplicate processing. • When the downstream is already overloaded: Retrying amplifies load → cascade failure. Combine with circuit breaker: open circuit instead of retrying when error rate is high. Retry budget: Limit total retries across all instances. If 100 pods each retry 3×, one failing service receives 300% of original traffic. Use retry budgets at the cluster level.

Saga compensation: When a step in a distributed Saga fails, execute compensating transactions on already-completed steps to undo their effects and restore consistency. Example — Order Saga: Step 1: Reserve inventory → Compensate: Release inventory reservation Step 2: Charge payment → Compensate: Issue refund Step 3: Schedule shipping → Compensate: Cancel shipment Step 4: Confirm order ← FAILS here Compensation sequence (reverse order): Cancel shipping request → Issue refund → Release inventory. Key property: Compensating transactions are themselves idempotent. Network failures during compensation → retry compensation, not the forward step. Orchestration implementation (recommended for complex Sagas): ```java // Temporal workflow @WorkflowImpl public class OrderSagaWorkflow { public void processOrder(Order order) { try { inventoryActivity.reserve(order); paymentActivity.charge(order); shippingActivity.schedule(order); } catch (Exception e) { // compensate in reverse order shippingActivity.cancel(order.getId()); paymentActivity.refund(order.getId()); inventoryActivity.release(order.getId()); } } } ``` Choreography implementation: Each service publishes a failure event. Downstream services subscribe to failure events and execute their own compensations. Challenges: • Some operations cannot be perfectly compensated: Email already sent, push notification delivered. Use "best effort" compensation (send cancellation email). • Compensation may also fail: Build retry into compensation steps. Track compensation status separately. • Concurrent sagas: One saga's compensation may conflict with another saga's forward progress. Use optimistic locking or reservation IDs to scope effects.

Multi-tenancy: A single running instance of a service (or system) serves multiple customers (tenants) with their data isolated from each other. Tenancy models: 1. Silo (full isolation): Each tenant has their own infrastructure — separate services, DBs, Kubernetes namespaces. Maximum isolation, maximum cost and complexity. Used for enterprise clients with compliance requirements. 2. Pool (shared everything): All tenants share the same DB tables. Tenant ID column on every table. Low cost, high density. Risk: one tenant's load affects others (noisy neighbor). Data isolation depends entirely on application-level filters. 3. Bridge (hybrid): Shared application layer, separate DB per tenant. Application code is shared (efficient), but data is physically isolated (compliance, performance isolation). Implementation — Pool model: • Every API request includes tenant context (JWT claim, subdomain, header) • TenantContext thread-local stores tenant ID for the request lifetime • Repository layer adds WHERE tenant_id = ? to every query automatically • DB-level row security (Postgres RLS): Policy enforces tenant isolation at DB level — backup defense Cross-service tenant propagation: Pass tenant ID in service-to-service calls via header (X-Tenant-ID). Each service extracts and applies it. Rate limiting per tenant: Apply separate rate limits so one tenant can't consume all capacity. Tenant onboarding in microservices: Tenant creation triggers events → each service creates tenant-specific configuration/data. Use an orchestrated onboarding workflow. Kubernetes namespaces for bridge model: One namespace per tier of tenant. Network policies enforce namespace isolation.

Strangler Fig pattern: Incrementally extract services from a monolith. New functionality is built as microservices; old functionality is progressively migrated out of the monolith until it can be retired. Step-by-step migration: Phase 1 — Identify boundaries: • Analyze the monolith for natural seams: different teams owning different modules, different scaling needs, different release cadences, bounded contexts in DDD terms. • Start with the least risky, most independent module (read-heavy, few dependencies). Phase 2 — Set up the routing layer: • Place an API gateway or reverse proxy in front of the monolith • All traffic still routes to monolith initially • Gateway will later route extracted modules to their new service Phase 3 — Extract the first service: • Create new service with its own DB (copy/migrate relevant data) • Run new service in parallel with monolith • Use a feature flag or small % traffic to verify the new service • Switch routing in gateway: /api/notifications → new NotificationService • Delete corresponding code from monolith Phase 4 — Database migration: • Monolith still writes to shared DB — new service reads from it initially • Gradually move writes: dual-write to both DBs → verify consistency → switch reads → stop writing to old schema • See expand-contract pattern for zero-downtime schema migration Phase 5 — Repeat until monolith is empty: • Each extraction reduces the monolith further • Extract high-churn modules next for maximum developer productivity gain Risks: Data consistency during dual-write phase. Distributed transactions where monolith and new service need to coordinate. Anti-corruption layer prevents monolith concepts from leaking into new service.

Design for failure: Assume every dependency will fail. Build services that degrade gracefully, recover quickly, and never cause cascading failures. Core principles: 1. Fail fast: Detect failures immediately rather than waiting for timeouts. Set aggressive timeouts on downstream calls (1-3 seconds, not 30 seconds). 2. Bulkhead isolation: Separate thread pools for different dependencies. If DB calls queue up and exhaust threads, HTTP calls to other services still work. Hystrix/Resilience4j Bulkhead. 3. Circuit breaker: After N failures in a window, stop calling the failing service (open circuit). Fail immediately. After cool-down, probe with one request (half-open). If successful, close circuit. 4. Graceful degradation: When a dependency is down, return a degraded but useful response. E.g., recommendation service down → return trending items instead of personalized recommendations. Notification service down → queue the notification for later instead of failing the order. 5. Idempotency: All operations must be safe to retry. Assign idempotency keys to requests — repeat the same key → same result, no duplicate effects. 6. Timeout everywhere: Every network call has an explicit timeout. No unbounded waits. 7. Health checks: Readiness probe → only route traffic to healthy pods. Liveness probe → restart pods that are stuck. 8. Retry with backoff: Retry transient failures with exponential backoff + jitter. But not for non-idempotent operations. 9. Dead letter queues: Messages that fail processing repeatedly go to DLQ for manual inspection — don't block the main queue. 10. Chaos engineering: Regularly inject failures (kill pods, add latency, drop connections) in staging to find weaknesses before production does.

Data mesh: An organizational and architectural paradigm where data ownership is distributed to domain teams, who treat their data as a product — discoverable, self-describing, and available to other teams. Counters the centralized data lake/warehouse model where a dedicated data team owns all data. Microservices connection: Microservices already mandate database-per-service (operational data ownership by domain). Data mesh extends this to analytical data — each domain also owns its analytical data products. Data mesh principles: 1. Domain data ownership: Order domain team owns and publishes Order analytics data. Not a central data engineering team. 2. Data as a product: Data products have SLOs (freshness, accuracy), documentation, schema versioned, accessible via self-serve interfaces (SQL, API, subscription). 3. Self-serve data infrastructure platform: Central platform team provides the infrastructure (data catalog, pipeline templates, storage) — not the data itself. 4. Federated computational governance: Global policies (PII handling, retention) enforced automatically across all data products. Implementation with microservices: • Each microservice publishes domain events to Kafka • Domain team owns a pipeline: Kafka → transformation → data product (Parquet in S3, or table in Snowflake) • Data catalog (DataHub, OpenMetadata) indexes all data products — discoverability • Other teams query the data product without coupling to the operational service Vs microservices data ownership (operational): Microservices own their transactional DB for reads/writes. Data mesh owns analytical snapshots/replicas for reporting and ML. Challenge: Teams need both software engineering AND data engineering skills. Platform team must make data pipeline creation self-service or it becomes a bottleneck.

Sidecar proxy: A proxy container deployed alongside every service container in the same pod (Kubernetes) or on the same host. Intercepts all inbound and outbound network traffic. Implements cross-cutting concerns transparently — without changing service code. Envoy Proxy: The most widely used sidecar proxy. Written in C++. Used by Istio, AWS App Mesh, and Consul Connect as the data plane. Envoy architecture: • Listener: Receives incoming connections on configured ports • Filter chain: Processes traffic through filters (HTTP, TCP, gRPC) • Router: Forwards to upstream clusters based on routing rules • Cluster: Represents a group of upstream service instances • Endpoint discovery: Dynamic discovery of upstream IPs via EDS (Endpoint Discovery Service) Features provided transparently: • Automatic retries: Retry failed requests based on configurable policy • Circuit breaking: Track error rate per upstream, open circuit on threshold • Load balancing: Round robin, least requests, ring hash, random — across upstream endpoints • TLS termination + mTLS: Encrypt all inter-service traffic, mutual auth • Observability: Automatic metrics (request count, latency histograms), traces (Zipkin/Jaeger), access logs — for every request, without code changes • Rate limiting: Integration with external rate limit service (Ratelimit) • Request routing: Header-based, weight-based, path-based routing Control plane (Istio): Istiod pushes configuration to all Envoy sidecars via xDS API (Listener Discovery Service, Route Discovery Service, Cluster Discovery Service, Endpoint Discovery Service). Operators define traffic policies in YAML; Istiod translates to Envoy config.

Inter-service authorization: Verifying not just that the caller is who it claims to be (authentication), but that it is allowed to perform the specific action on the specific resource (authorization). Authentication layer (who are you): • mTLS: Service presents a client certificate. The server verifies it. Proves the caller is a known service. Istio automates mTLS between all sidecars — zero code changes. • JWT service tokens: Service authenticates to identity provider (e.g., Vault, SPIFFE/SPIRE) and obtains a short-lived JWT. Includes claims: sub=order-service, scope=read:inventory. Target service validates JWT signature. Authorization layer (what can you do): • Scope-based: JWT contains scopes. Service checks required scope before executing operation. order-service must have scope=read:inventory to call inventory-service GET endpoint. • RBAC for services: Each service has a service account with assigned roles. Open Policy Agent (OPA) evaluates policy: can service account order-service call inventory-service/reserve-stock? Yes, it has the inventory-writer role. • Network policies (Kubernetes): NetworkPolicy resources restrict which pods can connect to which other pods on which ports. First line of defense — block unexpected connections at network level. Open Policy Agent (OPA): • Centralized policy engine. Services call OPA with request context: {caller: "order-service", action: "POST", resource: "/inventory/reserve"} • OPA evaluates Rego policy rules and returns allow/deny • Policies stored in git, versioned, deployed to OPA Practical approach for most teams: mTLS (via Istio) for service authentication + JWT scopes for coarse-grained authorization + NetworkPolicy as baseline. Full OPA for fine-grained, complex authorization requirements.

Microservices are powerful but not universally correct. Know when they hurt more than they help. Do NOT use microservices when: 1. Small team (< 5-8 engineers): Microservices require operational maturity — CI/CD, container orchestration, distributed tracing, service discovery. A 3-person team will spend more time on infrastructure than product. 2. Unclear domain boundaries: If you can't clearly define what each service is responsible for, you'll end up with a distributed monolith — tightly coupled services that deploy independently but break each other anyway. 3. Early-stage startup / uncertain domain: When the product is changing fast and you're still discovering what you're building, microservices lock you into premature boundaries. Refactoring service boundaries is painful. Refactoring a monolith module is a rename. 4. CRUD-heavy applications with no scaling differentiation: If all components have similar load and a PostgreSQL monolith handles it, microservices add zero value and enormous complexity. 5. Strong data consistency requirements everywhere: If almost every operation needs strong ACID consistency across multiple entities, microservices force you into Saga complexity for no benefit. Monolith + single DB handles transactions trivially. 6. No DevOps/platform team: Microservices multiply operational surface area — each service needs its own monitoring, alerts, deployment pipeline, secrets management. Without platform infrastructure, this collapses into chaos. Better starting point: Modular monolith — enforce module boundaries within a single deployable unit. Extract microservices when you have specific reasons: team autonomy, independent scaling, different technology requirements, or the monolith's build/test/deploy times are slowing you down. Measure the pain before treating it with microservices.