Chaos Engineering

// Chaos engineering process
//
// 1. Define steady state
//    "Order API returns 200 in < 200ms for 99.9% of requests"
//
// 2. Hypothesise
//    "If we kill 2 of 5 order-service pods, Kubernetes reschedules
//     them within 30s and latency stays below 200ms because the
//     remaining 3 pods absorb the traffic"
//
// 3. Run experiment
//    Kill 2 pods at 10:00 AM
//    Monitor: request rate, error rate, p99 latency
//
// 4. Observe results
//    ✅ Pods rescheduled in 15s
//    ⚠️ Latency spiked to 350ms for 20 seconds (unexpected)
//    ✅ No 5xx errors
//
// 5. Learn and improve
//    Finding: HPA takes 30s to scale up → pre-warm extra pods
//    Action: Set minReplicas from 3 to 5

// Blast radius controls:
// Start in staging → move to production
// Start with non-critical services → then critical
// Set maximum impact: "kill at most 1 pod per service"
// Have an abort switch: stop experiment immediately if SLO breached

// Litmus Chaos — Kubernetes-native chaos experiments
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-chaos
spec:
  appinfo:
    appns: default
    applabel: app=order-service
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'       # chaos for 60 seconds
            - name: CHAOS_INTERVAL
              value: '10'       # delete a pod every 10s
            - name: FORCE
              value: 'true'

// Common chaos experiments:
// 🔴 Pod kill:       Kill random pods → test auto-recovery
// 🟡 Network latency: Add 500ms latency → test timeouts and circuit breakers
// 🟡 Network partition: Isolate a service → test fallbacks
// 🔴 DNS failure:    Block DNS → test service discovery resilience
// 🟡 CPU stress:     Max out CPU → test auto-scaling and throttling
// 🔴 Disk full:      Fill disk → test logging and data handling
// 🟡 Zone failure:   Kill all pods in one AZ → test multi-AZ resilience

// Netflix Chaos Monkey — random instance termination
// Runs during business hours in production
// Forces engineers to build resilient services
// "If you can't survive Chaos Monkey, you can't survive production"