Kubernetes — Cheat Sheet

Kubernetes (K8s) is an open-source container orchestration platform that automates deployment, scaling, and management of containerized applications. Problems it solves: • Self-healing: Restarts failed containers, replaces unresponsive nodes • Auto-scaling: Scales pods up/down based on CPU/memory metrics or custom metrics • Service discovery & load balancing: Built-in DNS and traffic distribution • Rolling updates & rollbacks: Zero-downtime deployments with automatic rollback on failure • Resource management: Bin-packing containers on nodes based on resource requests/limits • Configuration management: ConfigMaps and Secrets for externalizing config Without Kubernetes: You'd manually manage where containers run, handle failures manually, and script scaling yourself.

A Pod is the smallest deployable unit in Kubernetes. It wraps one or more tightly coupled containers that share: • Network namespace: Same IP address, can communicate via localhost • Storage volumes: Shared volumes mounted in multiple containers • Lifecycle: Start and stop together Why not just containers? Some applications need helper containers (sidecars) to work: an Envoy proxy, a log forwarder, a config sync agent. These must share network and storage with the main container. In practice: Most pods have a single container. Multi-container pods are the exception (sidecar, adapter, ambassador patterns). Pods are ephemeral — they can be killed and replaced on any node. Never rely on a Pod's IP address directly; use Services.

Deployment: Manages stateless pods. Pods are interchangeable — any pod can replace any other. Supports rolling updates, rollbacks, and scaling. Pods get random names (web-abc123). Use for stateless services (web servers, APIs). StatefulSet: Manages stateful pods that need stable identity. Each pod gets a predictable, stable name (postgres-0, postgres-1). Pods start/stop in order. Each pod has its own PersistentVolumeClaim that persists across restarts. Use for databases, Kafka, Zookeeper, Elasticsearch. Key StatefulSet guarantees: • Stable network identifiers (pod-name.service-name.namespace.svc.cluster.local) • Stable persistent storage (PVC not deleted when pod is replaced) • Ordered, graceful deployment and scaling

Services provide stable network endpoints for pods (pods come and go, Services stay). • ClusterIP (default): Exposes the service on an internal cluster IP. Only reachable within the cluster. Used for internal service-to-service communication. • NodePort: Exposes the service on a static port on every node's IP. Reachable from outside the cluster via <NodeIP>:<NodePort>. Useful for development, not production. • LoadBalancer: Provisions a cloud provider load balancer (AWS ELB, GCP LB). Assigns an external IP. Standard way to expose services in cloud environments. • ExternalName: Maps the service to a DNS name (e.g., an external database). No proxying — just DNS. • Headless (clusterIP: None): Returns pod IPs directly instead of a virtual IP. Used by StatefulSets for direct pod addressing.

Both are health checks that Kubernetes runs against pods. Readiness probe: Determines if the pod is ready to receive traffic. If it fails, the pod is removed from the Service's endpoint list — no traffic is routed to it. Use for: slow startup, temporary unavailability (cache warming, DB migration running). Liveness probe: Determines if the pod is alive. If it fails repeatedly, Kubernetes restarts the container. Use for: detecting deadlocks, frozen processes that are still running but unresponsive. Startup probe: Delays liveness/readiness probes until the startup probe succeeds. Prevents slow-starting applications from being killed before they initialize. Probe types: httpGet (HTTP check), exec (run a command), tcpSocket (TCP connection check). Best practice: Always define both. Set readiness thresholds conservative; liveness thresholds generous.

Kubernetes offers multiple autoscaling mechanisms: HPA (Horizontal Pod Autoscaler): Adjusts the number of pod replicas based on metrics. Default metrics: CPU and memory utilization. Custom metrics via Prometheus Adapter (request rate, queue depth). Checks every 15 seconds by default. VPA (Vertical Pod Autoscaler): Automatically adjusts CPU/memory requests and limits for pods. Useful for right-sizing. Requires pod restarts to apply — not suitable for all workloads. KEDA (Kubernetes Event-Driven Autoscaler): Scales based on external event sources (Kafka consumer lag, SQS queue depth, Redis list length). Scales to zero when no events. Community extension, widely adopted. Cluster Autoscaler: Adds or removes nodes when pods can't be scheduled (insufficient resources) or nodes are underutilized.

RBAC (Role-Based Access Control) restricts who can do what in a Kubernetes cluster. Key objects: • Role: Defines permissions (verbs on resources) within a namespace. Example: allow get, list, watch on pods. • ClusterRole: Like Role but cluster-wide (across all namespaces). • RoleBinding: Binds a Role to a subject (user, group, or ServiceAccount) within a namespace. • ClusterRoleBinding: Binds a ClusterRole to a subject cluster-wide. Subjects: Users (human operators), Groups, ServiceAccounts (pod identities). Best practice: Principle of least privilege — give only the permissions needed. Use ServiceAccounts for pod identities (not admin credentials). Avoid ClusterRoleBindings unless truly needed. Audit regularly with kubectl auth can-i.

ConfigMap: Stores non-sensitive key-value configuration (database hostnames, feature flags, application.properties content). Can be consumed as environment variables or mounted as files. Secret: Stores sensitive data (passwords, tokens, TLS certs). Values are base64-encoded in etcd (not encrypted by default). Enable EncryptionConfiguration or use external secret management (External Secrets Operator with AWS Secrets Manager/Vault) for real encryption at rest. Both can be injected as: • Environment variables: Simple but requires pod restart to pick up changes • Volume mounts: Files appear in the container. Secret changes propagate automatically (within ~1 minute) without restart when mounted as volumes Best practice: Treat Secrets as sensitive. Enable KMS encryption, limit RBAC access, avoid logging secret values, use short-lived credentials with rotation.

Rolling updates update a Deployment incrementally — replacing old pods with new ones, maintaining availability throughout. Key fields in Deployment spec: • maxSurge: Maximum pods above desired count during rollout (e.g., 25% or 1) • maxUnavailable: Maximum pods below desired count during rollout (e.g., 25% or 0 for zero-downtime) Process: 1. Create new ReplicaSet with new image 2. Scale up new RS by maxSurge 3. Scale down old RS by maxUnavailable 4. Repeat until all pods are updated 5. Old RS kept with 0 replicas (enables rollback) Rollback: kubectl rollout undo deployment/my-app Status: kubectl rollout status deployment/my-app Pre-requisite for zero-downtime: readiness probes must be configured — Kubernetes waits for new pods to become ready before removing old ones.

A DaemonSet ensures exactly one pod runs on each node (or a subset matching a nodeSelector). When a new node joins the cluster, a pod is automatically scheduled on it. When a node is removed, the pod is garbage collected. Use cases: • Logging agents: Fluentd or Filebeat to collect logs from every node • Monitoring agents: Prometheus Node Exporter to collect node-level metrics • Network plugins: CNI plugins (Calico, Flannel) run as DaemonSets • Security agents: Falco for container runtime security monitoring • Storage plugins: Ceph or Gluster storage daemons DaemonSets run on all nodes by default. Use nodeSelector or tolerations to restrict to specific nodes (e.g., only GPU nodes, only worker nodes).

PersistentVolume (PV): A piece of storage in the cluster provisioned by an admin or dynamically by a StorageClass. It's a cluster resource independent of any pod lifecycle. Backed by: NFS, iSCSI, cloud volumes (EBS, GCE PD, Azure Disk), etc. PersistentVolumeClaim (PVC): A request for storage by a user. Specifies size, access mode, and StorageClass. Kubernetes binds the PVC to a matching PV. Access modes: • ReadWriteOnce (RWO): Mounted by one node for read/write • ReadOnlyMany (ROX): Mounted by many nodes for reading • ReadWriteMany (RWX): Mounted by many nodes for read/write (NFS, EFS) StorageClass: Enables dynamic provisioning — PVC is created, Kubernetes automatically provisions a PV. No pre-provisioning needed. Define different classes for SSD, HDD, backup storage.

Helm is the Kubernetes package manager. A Helm chart is a collection of templates that define a set of Kubernetes resources. Charts are parameterized — values.yaml provides configurable defaults. Structure: chart/ Chart.yaml — metadata (name, version, description) values.yaml — default configuration values templates/ — Kubernetes manifest templates with Go templating Key commands: helm install my-release chart/ — deploy a chart helm upgrade my-release chart/ --set image.tag=2.0 helm rollback my-release 1 helm list — list installed releases Benefits: Reusable deployments, environment-specific overrides (values-prod.yaml), release versioning and rollback, dependency management (sub-charts). Artifact Hub: Public repository of community charts (PostgreSQL, Kafka, Redis).

A Namespace is a virtual cluster within a Kubernetes cluster. Resources (pods, services, configmaps) are scoped to a namespace. Different namespaces can have resources with the same name. Default namespaces: default, kube-system (Kubernetes internals), kube-public (public cluster info), kube-node-lease. Why use multiple namespaces: • Environment separation: dev, staging, prod in one cluster (cost-effective for small teams) • Team isolation: Separate namespaces per team with RBAC policies • Resource quotas: Limit CPU/memory per namespace (ResourceQuota) • Network policies: Restrict cross-namespace communication Namespaces don't provide node-level isolation — a misbehaving pod can still affect others. For strong isolation, use separate clusters. DNS format: service-name.namespace.svc.cluster.local

LoadBalancer Service: One cloud load balancer per service. Expensive — 10 services = 10 load balancers. No path-based routing. Ingress: An API object that manages external HTTP/HTTPS access to services. A single Ingress Controller (nginx, Traefik, AWS ALB, Istio) handles all routing. Ingress enables: • Host-based routing: api.example.com → API Service • Path-based routing: /v1 → v1 Service, /v2 → v2 Service • TLS termination: SSL certificate managed centrally (cert-manager + Let's Encrypt) • Middleware: Rate limiting, auth, headers — configured in annotations Ingress Controller: Reads Ingress objects and configures the actual load balancer/reverse proxy. Must be deployed separately. For production: Use one Ingress Controller with many Ingress rules instead of many LoadBalancer Services.

Resource requests and limits control CPU and memory allocation for containers. Requests: The minimum resources guaranteed to the container. Used by the scheduler to find a suitable node (node must have at least this much free). If no request is set, the scheduler may place it anywhere, potentially causing OOM or CPU starvation. Limits: The maximum resources a container can use. For memory: exceeding the limit causes OOM kill and container restart. For CPU: throttled (not killed). Best practice: Always set both. Start with requests = limits (Guaranteed QoS class). Right-size with VPA recommendations. QoS classes (affect eviction priority under node pressure): • Guaranteed: requests == limits for all containers • Burstable: limits > requests • BestEffort: No requests or limits set (evicted first) Namespace defaults: Use LimitRange to set defaults; ResourceQuota to cap total namespace resource usage.

The control plane manages the cluster state and orchestrates workloads. It runs on master/control-plane nodes. kube-apiserver: • Front-end for the Kubernetes API — all communication goes through it • Validates and processes REST requests • Reads/writes cluster state to etcd • Handles authentication, authorization (RBAC), and admission control • Horizontally scalable — multiple instances for HA etcd: • Distributed key-value store — the single source of truth for cluster state • Stores all Kubernetes object definitions • Uses Raft consensus for consistency • Must be backed up regularly kube-scheduler: • Watches for unscheduled pods and assigns them to nodes • Considers: resource requests/limits, affinity/anti-affinity, taints/tolerations, node conditions • Pluggable — custom schedulers and scheduler framework plugins kube-controller-manager: • Runs all built-in controllers in a single process • Node Controller: Detects and responds to node failures • ReplicaSet Controller: Ensures desired pod count • Deployment Controller: Manages rolling updates • Job Controller: Tracks completion of Jobs • Endpoint Controller: Populates Endpoints objects (links Services to Pods) cloud-controller-manager: • Cloud-specific control loops • Node lifecycle: Integrates cloud node provisioning • Route controller: Configures cloud network routes • Service controller: Creates cloud load balancers for LoadBalancer Services HA control plane: Run multiple API servers behind a load balancer. etcd cluster with odd number of nodes (3, 5). Leader election for controllers and scheduler.

etcd: A distributed, strongly consistent key-value store. The single source of truth for all Kubernetes cluster state — every object (pods, deployments, services, secrets, config maps) is stored in etcd. Why critical: If etcd loses data, the entire cluster configuration is lost — you can't recover without a backup. etcd is the only stateful component in the control plane. etcd internals: • Raft consensus algorithm: All writes go through a leader, replicated to followers before acknowledging. Requires majority quorum (n/2 + 1) for writes. • Even node count: Always use odd number (3, 5, 7). With 3 nodes: tolerates 1 failure. With 5 nodes: tolerates 2 failures. • Watch API: Clients register watches — etcd pushes change notifications. This is how the API server propagates changes to controllers and kubelet. Production etcd setup: • Dedicated nodes: Don't co-locate etcd with other workloads — I/O intensive • Fast SSD: etcd is latency-sensitive — NVMe SSDs recommended • Separate etcd cluster: Control plane etcd stacked vs external etcd topology • Regular backups: Snapshot every hour + store in S3 Backup: ```bash etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/etcd/ca.crt \ --cert=/etc/etcd/etcd.crt \ --key=/etc/etcd/etcd.key ``` Restore: ```bash etcdctl snapshot restore /backup/etcd.db --data-dir=/var/lib/etcd-restore ``` Encryption: Enable EncryptionConfiguration to encrypt Secrets at rest in etcd.

The Kubernetes scheduler assigns unscheduled pods to nodes through a filtering and scoring process. Scheduling cycle: 1. Watch: Scheduler watches for pods with nodeName empty (unscheduled) 2. Filter (predicates): Eliminate nodes that can't run the pod 3. Score (priorities): Rank remaining nodes 4. Assign: Pod gets nodeName set to the highest-scored node 5. Binding: API server updates the pod object Filtering checks: • NodeResourcesFit: Node has sufficient CPU and memory for pod requests • NodeAffinity: Node matches pod's node selector/affinity rules • TaintToleration: Pod tolerates all node taints • NodeUnschedulable: Node not marked as unschedulable • PodTopologySpread: Pod placement satisfies topology spread constraints • VolumeBinding: Required PVCs can be bound on this node • HostPort: No port conflicts Scoring functions: • LeastAllocated: Prefer nodes with more free resources (spread workload) • MostAllocated: Pack pods tightly (bin-packing, reduce node count) • ImageLocality: Prefer nodes that already have the container image • NodeAffinity: Higher score for preferred affinity rules • PodTopologySpread: Spread pods across failure domains Scheduler framework: Extensible plugin system. Add custom filtering and scoring plugins without modifying scheduler code. Example: schedule pods only on nodes with a specific hardware feature (GPU type, FPGA). Pod disruption: • kube-scheduler and kube-controller-manager both use leader election — only one instance active at a time. Others are standby. If leader dies, new election in seconds. Manual scheduling: Set pod.spec.nodeName directly — bypasses scheduler entirely.

Controller: A control loop that watches the current state of the cluster and takes actions to move it toward the desired state. The fundamental pattern behind all Kubernetes automation. Control loop pseudocode: ``` for { desired = getDesiredState() // from etcd via API server actual = getActualState() // observe real world if desired != actual { reconcile(desired, actual) // take action to converge } sleep(or wait for change notification) } ``` Examples of built-in controllers: • ReplicaSet controller: Desired = 5 replicas. Actual = 3. Action: create 2 pods. • Deployment controller: Desired = new image version. Actual = old image. Action: rolling update. • Node controller: Desired = all nodes healthy. Node stops heartbeating. Action: mark pods NotReady, evict after timeout. • Job controller: Job spec says run 10 times. 7 completed. Action: create 3 more pods. Controller watching mechanism: • Controllers don't poll — they use the Kubernetes Watch API • API server sends change events (add/modify/delete) to watching controllers • Informers: Client-side caching layer for watch events. Reduces API server load. • Work queue: Events go into a queue. Controller processes events from queue. Retries on failure. Custom controllers (Operators): Extend Kubernetes with domain-specific automation. Watch custom resources (CRDs) and take business-specific actions. Idempotency: Controllers are designed to be idempotent — running the reconcile loop multiple times with the same state produces the same result.

kubelet: The primary node agent running on every Kubernetes worker node. Responsible for making the node functional as part of the cluster. kubelet responsibilities: 1. Pod management: Watches the API server for pods assigned to its node. Creates, updates, and deletes containers via the Container Runtime Interface (CRI). 2. Health monitoring: Runs liveness and readiness probes. Restarts unhealthy containers. 3. Resource reporting: Reports node capacity (CPU, memory, storage) and usage to the API server. 4. Volume management: Mounts/unmounts volumes and PVCs for pods. 5. Node registration: Registers the node with the API server on startup. 6. Status updates: Sends NodeStatus and PodStatus updates to the API server periodically. kubelet + CRI: kubelet doesn't manage containers directly — it communicates via CRI (Container Runtime Interface). CRI implementations: • containerd: Default runtime for most Kubernetes distributions • CRI-O: Lightweight CRI implementation kubelet → CRI (gRPC) → containerd → containerd-shim → runc → container kubelet configuration: ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd maxPods: 110 kubeReserved: cpu: 200m memory: 250Mi systemReserved: cpu: 200m memory: 250Mi evictionHard: memory.available: "100Mi" nodefs.available: "10%" ``` kube-proxy: The other node component. Maintains network rules (iptables or IPVS) for Service ClusterIP routing. Every Service IP maps to actual pod IPs via iptables DNAT rules. kube-proxy modes: • iptables (default): Rules evaluated linearly — O(n) for n services • IPVS: Hash-based — O(1) lookup. Better for large clusters with many services • eBPF (Cilium): Bypass kernel networking stack entirely — highest performance

CNI (Container Network Interface): A standard for configuring network interfaces in Linux containers. When kubelet creates a pod, it calls the CNI plugin to set up networking for the pod. Pod networking requirements: • Every pod gets a unique IP address • Pods on the same node can communicate directly • Pods on different nodes can communicate without NAT • Nodes can communicate with pods directly How CNI works: 1. kubelet creates pod network namespace 2. kubelet calls CNI plugin binary (e.g., /opt/cni/bin/calico) 3. CNI plugin creates a veth pair: one end in pod namespace (eth0), one end on host 4. CNI plugin assigns IP to pod's eth0 from the pod CIDR 5. CNI plugin configures routing so other pods can reach this IP Popular CNI plugins: Calico: • BGP routing: Each node advertises its pod CIDR to others via BGP • No overlay network needed (in native mode) — highly performant • Rich network policy support • Can use VXLAN overlay for environments without BGP support Flannel: • Simple VXLAN overlay • All pod traffic encapsulated in UDP packets between nodes • Easy to deploy, less performant than Calico Cilium: • eBPF-based: Bypasses iptables and kernel networking stack • Extreme performance (no iptables rules) • Identity-based network policies (pod labels, not IPs) • Hubble: Built-in network observability • L7 network policies (HTTP path/method level) Pod CIDR: Each node gets a subnet (e.g., 10.0.1.0/24). Cluster pod CIDR is 10.0.0.0/16. Routing ensures cross-node pod communication.

Network Policy: A Kubernetes resource that controls which pods can communicate with each other and with external endpoints. By default, Kubernetes allows all pod-to-pod communication — Network Policies add restrictions. Default behavior: Without Network Policies, all pods in all namespaces can communicate freely. Network Policy basics: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-api-to-db namespace: production spec: podSelector: matchLabels: role: database # applies to db pods policyTypes: [Ingress] # restrict incoming traffic ingress: - from: - podSelector: matchLabels: role: api # allow only from api pods - namespaceSelector: matchLabels: name: monitoring # and monitoring namespace ports: - protocol: TCP port: 5432 ``` Default deny all pattern: ```yaml # Step 1: Deny all ingress spec: podSelector: {} # all pods in namespace policyTypes: [Ingress] # no ingress rules = deny all # Step 2: Explicitly allow what's needed ``` Egress policies: Restrict outbound traffic from pods — prevent data exfiltration, restrict pods to only talk to expected services. Limitations: • CNI plugin must support Network Policies (Calico, Cilium, Weave — yes; Flannel — no) • Network Policies select pods by labels — dynamic IPs, not static • No deny rules — you can only allow, and absence of allow = deny • Layer 3/4 only (IP, port) — use Istio for L7 policies (HTTP path/method) Zero-trust networking: Default deny all + explicit allow. Namespace isolation. Separate network policies per microservice.

Taints and tolerations work together to ensure pods are not scheduled on inappropriate nodes. Taint: Applied to a node. Repels pods that don't explicitly tolerate it. Toleration: Applied to a pod. Allows scheduling on a tainted node. Taint effects: • NoSchedule: New pods without toleration won't be scheduled here. Existing pods not affected. • PreferNoSchedule: Scheduler tries to avoid scheduling here, but not guaranteed. • NoExecute: New pods without toleration won't be scheduled. Existing pods without toleration are evicted. Commands: ```bash # Add taint to node kubectl taint nodes node1 key=value:NoSchedule kubectl taint nodes node1 gpu=true:NoSchedule # Remove taint kubectl taint nodes node1 key:NoSchedule- ``` Pod toleration: ```yaml spec: tolerations: - key: gpu operator: Equal value: "true" effect: NoSchedule - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300 # stay for 5 min after node becomes not-ready ``` Common use cases: • Dedicated nodes: Taint GPU nodes — only GPU-needing pods schedule there • Master nodes: Tainted node-role.kubernetes.io/control-plane:NoSchedule — prevents regular pods on control plane • Node maintenance: Taint a node NoExecute to drain it gracefully • Spot/preemptible nodes: Taint with spot=true:NoSchedule — only tolerant pods run there Vs node affinity: Taints/tolerations work from the node's perspective (reject non-tolerating pods). Node affinity works from the pod's perspective (pods prefer/require certain nodes). Used together for precise placement.

Affinity rules influence where pods get scheduled — working from the pod's perspective (unlike taints which work from the node's perspective). Node Affinity: Constrains which nodes a pod can be scheduled on based on node labels. Types: • requiredDuringSchedulingIgnoredDuringExecution: Hard requirement — pod won't schedule if not met • preferredDuringSchedulingIgnoredDuringExecution: Soft preference — scheduler tries but not required ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: [amd64, arm64] - key: node.kubernetes.io/instance-type operator: NotIn values: [t2.micro] # avoid cheapest instances preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: topology.kubernetes.io/zone operator: In values: [us-east-1a] # prefer zone a ``` Pod Affinity: Schedule pods near (or away from) other pods with specific labels. ```yaml affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: redis # must be on same node as Redis topologyKey: kubernetes.io/hostname podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: api # must NOT be on same node as other API pods topologyKey: kubernetes.io/hostname ``` Pod anti-affinity use cases: • High availability: Spread replicas across different nodes/zones — one node failure doesn't take all replicas • Co-location for performance: App pods near cache pods on same node — no network hop topologyKey: The node label key defining the scope. kubernetes.io/hostname = node, topology.kubernetes.io/zone = availability zone, topology.kubernetes.io/region = region.

Job: Runs one or more pods to completion. Unlike Deployments (keep pods running), Jobs run pods until they successfully finish. ```yaml apiVersion: batch/v1 kind: Job metadata: name: db-migration spec: completions: 1 # total successful completions needed parallelism: 1 # pods running in parallel backoffLimit: 4 # retry limit on failure activeDeadlineSeconds: 600 # timeout template: spec: restartPolicy: OnFailure # Never or OnFailure (not Always) containers: - name: migrate image: myapp:1.0 command: [java, -jar, app.jar, --run-migrations] ``` Job patterns: • Single completion: completions=1, parallelism=1 — run once • Fixed completions: completions=10, parallelism=2 — 10 completions, 2 at a time • Work queue: completions unset, parallelism=3 — workers consume from queue, stop when queue empty CronJob: Schedules Jobs on a time-based schedule (like Unix cron). ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: daily-report spec: schedule: "0 2 * * *" # 2am daily (cron syntax) timeZone: "Asia/Kolkata" concurrencyPolicy: Forbid # Forbid | Allow | Replace startingDeadlineSeconds: 300 successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 1 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: reporter image: report-generator:latest ``` concurrencyPolicy: • Forbid: Skip new run if previous still running • Allow: Run concurrently (default) • Replace: Cancel previous, start new Use cases: Database migrations, scheduled reports, cleanup jobs, batch processing, sending scheduled notifications.

ServiceAccount: An identity for processes running in pods. When a pod needs to call the Kubernetes API (or other services using IRSA/Workload Identity), it authenticates using its ServiceAccount. Default behavior: Every pod gets the default ServiceAccount in its namespace if none is specified. Default SA has minimal permissions (or none in restrictive setups). Creating and using a ServiceAccount: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: order-service-sa namespace: production annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/order-service-role # IRSA --- apiVersion: apps/v1 kind: Deployment spec: template: spec: serviceAccountName: order-service-sa automountServiceAccountToken: false # disable if not needed ``` SA token: Kubernetes automatically mounts a token at /var/run/secrets/kubernetes.io/serviceaccount/token (projected volume). Pod uses this to authenticate to the Kubernetes API. RBAC binding to ServiceAccount: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: order-service-binding namespace: production subjects: - kind: ServiceAccount name: order-service-sa namespace: production roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` IRSA (IAM Roles for Service Accounts) on AWS EKS: • Annotate SA with IAM role ARN • Pod automatically gets AWS credentials via projected token + OIDC • No long-lived credentials in containers Workload Identity on GKE: Similar pattern — maps Kubernetes SA to GCP Service Account. No credential management needed.

Admission controllers: Plugins that intercept requests to the Kubernetes API server after authentication and authorization, but before objects are persisted to etcd. Can validate and/or mutate objects. Two phases: 1. Mutating admission: Can modify the object (add labels, inject sidecars, set defaults) 2. Validating admission: Can approve or reject the object (enforce policies) Built-in admission controllers: • NamespaceLifecycle: Prevents creation in terminating namespaces • LimitRanger: Applies LimitRange defaults to pods without resource limits • ResourceQuota: Enforces namespace resource quotas • DefaultStorageClass: Assigns default StorageClass to PVCs • PodSecurity: Enforces Pod Security Standards • ServiceAccount: Automounts SA tokens Admission Webhooks: Custom admission controllers via HTTP webhooks. Two types: MutatingAdmissionWebhook: ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: sidecar-injector webhooks: - name: inject-sidecar.example.com rules: - apiGroups: [""] apiVersions: [v1] resources: [pods] operations: [CREATE] clientConfig: service: name: sidecar-injector namespace: kube-system path: /mutate caBundle: <base64-CA> admissionReviewVersions: [v1] sideEffects: None failurePolicy: Fail # or Ignore ``` ValidatingAdmissionWebhook: Same structure but enforces policies. Use cases: • Istio sidecar injection: Automatically injects Envoy sidecar via MutatingWebhook • Policy enforcement (OPA Gatekeeper, Kyverno): Validates resources against custom policies • Default injection: Auto-add labels, annotations, security contexts • Image policy: Block images from untrusted registries

CRD (Custom Resource Definition): Extends the Kubernetes API with custom resource types. After creating a CRD, you can create instances of the custom resource just like built-in resources (Pods, Services). Creating a CRD: ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: databases.example.com spec: group: example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: engine: type: string enum: [postgres, mysql] size: type: string replicas: type: integer minimum: 1 scope: Namespaced # or Cluster names: plural: databases singular: database kind: Database shortNames: [db] ``` Using the custom resource: ```yaml apiVersion: example.com/v1 kind: Database metadata: name: orders-db namespace: production spec: engine: postgres size: large replicas: 3 ``` ```bash kubectl get databases kubectl describe database orders-db ``` CRD validation: OpenAPI v3 schema validates custom resource fields at admission time. CRD versioning: Support multiple versions with conversion webhooks for migration between v1 and v2. CRD + Controller = Operator: CRD defines the desired state schema. A custom controller watches for CRD instances and reconciles actual state. Together they form the Operator pattern — domain-specific automation built on Kubernetes primitives.

Operator: A Kubernetes application that uses CRDs + custom controllers to encode operational knowledge about a specific software domain. Automates tasks a human operator would do — provisioning, configuration, scaling, backups, upgrades. Operator pattern components: 1. CRD: Defines the domain-specific resource (e.g., ElasticsearchCluster, PostgresDatabase, KafkaCluster) 2. Custom Controller: Watches the CRD and reconciles the actual state with the desired state 3. Operational knowledge: Business logic about how to deploy, scale, backup, and recover the software Example — PostgreSQL Operator (CloudNativePG): ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: pg-orders spec: instances: 3 # 1 primary + 2 replicas storage: size: 100Gi backup: barmanObjectStore: destinationPath: s3://my-bucket/pg-orders s3Credentials: {accessKeyId: {...}, secretAccessKey: {...}} ``` Operator handles: primary election, streaming replication, automatic failover, backups, SSL, monitoring setup. Operator capabilities (maturity levels): 1. Basic install: Automate installation and configuration 2. Seamless upgrades: Manage minor/major version upgrades 3. Full lifecycle: Backup/restore, failure recovery 4. Deep insights: Expose metrics, alerts, dashboards 5. Auto pilot: Automatic tuning, anomaly detection, auto-remediation Building operators: • Operator SDK: Kubebuilder-based scaffold (Go) • Kubebuilder: Lower-level framework • KOPF (Python): Kubernetes Operator Pythonic Framework • Shell-operator: Simple operators in shell scripts Well-known operators: cert-manager, Prometheus Operator, Elastic Cloud on Kubernetes, Strimzi (Kafka), CockroachDB Operator, Rook (Ceph storage).

Pod Security Standards (PSS): A Kubernetes-native framework (replacing deprecated PodSecurityPolicy) that defines security profiles for pods. Enforced via the PodSecurity admission controller (built-in since K8s 1.25). Three security profiles: Privileged: No restrictions. Allows everything including privileged containers, host network, any capabilities. For trusted system pods (monitoring agents, CNI plugins). Baseline: Minimally restrictive. Prevents known privilege escalation. Allows: • Non-privileged containers • Most capabilities • hostPath volumes with restrictions Blocks: privileged containers, hostPID/hostNetwork, dangerous capabilities (NET_RAW, SYS_ADMIN), seccomp:unconfined. Restricted: Heavily restricted following security best practices. Requires: • Non-root user (runAsNonRoot: true) • Non-privileged container • Drop ALL capabilities • No privilege escalation (allowPrivilegeEscalation: false) • Seccomp profile: RuntimeDefault or Localhost • Read-only root filesystem recommended Applied at namespace level: ```yaml apiVersion: v1 kind: Namespace metadata: name: production labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: v1.28 pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/audit: restricted ``` Modes: • enforce: Reject pods that violate the policy • warn: Allow but warn in API response • audit: Allow but record in audit log Pod security context for restricted profile: ```yaml spec: securityContext: runAsNonRoot: true runAsUser: 1001 seccompProfile: type: RuntimeDefault containers: - securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] readOnlyRootFilesystem: true ```

By default, Kubernetes Secrets are base64-encoded but NOT encrypted in etcd. Anyone with etcd access can decode them trivially. Encryption at rest adds a real encryption layer. Built-in encryption (EncryptionConfiguration): ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: [secrets, configmaps] providers: - aescbc: # AES-CBC encryption (256-bit key) keys: - name: key1 secret: <base64-encoded-32-byte-key> - identity: {} # fallback: plaintext (for migration) ``` Apply: Add --encryption-provider-config to kube-apiserver. New secrets are encrypted. Existing secrets: kubectl get secrets -A -o json | kubectl replace -f - to re-encrypt all. KMS provider (recommended for production): ```yaml - kms: apiVersion: v2 name: myKMSPlugin endpoint: unix:///tmp/kms.socket cachesize: 1000 timeout: 3s ``` KMS plugin integrates with AWS KMS, GCP Cloud KMS, Azure Key Vault, or HashiCorp Vault. Data encryption key (DEK) is generated per secret, encrypted with the KMS master key. DEK stored alongside secret in etcd. Decryption requires KMS service. External Secrets Operator (ESO): ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-password spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-password # creates a Kubernetes Secret data: - secretKey: password remoteRef: key: prod/db/password property: DB_PASSWORD ``` Secrets never stored in Kubernetes etcd — fetched dynamically from AWS Secrets Manager, GCP Secret Manager, or Vault. Rotation: ESO re-syncs on refresh interval.

cert-manager: A Kubernetes-native certificate management controller. Automates the issuance and renewal of TLS certificates from various sources (Let's Encrypt, Vault, self-signed CA, internal CAs). Core resources: • Issuer / ClusterIssuer: Defines certificate authority configuration (ACME, Vault, CA) • Certificate: Defines a desired TLS certificate • CertificateRequest: Internal resource tracking a certificate request Let's Encrypt (ACME) setup: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-account-key solvers: - http01: ingress: class: nginx # HTTP-01 challenge via ingress - dns01: route53: # DNS-01 challenge via Route53 (for wildcards) region: us-east-1 ``` Requesting a certificate: ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: api-tls namespace: production spec: secretName: api-tls-secret # stores the issued certificate issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames: - api.example.com - www.api.example.com duration: 2160h # 90 days renewBefore: 360h # renew 15 days before expiry ``` cert-manager automatically renews certificates before expiry. Ingress annotation (auto-certificate): ```yaml annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: tls: - hosts: [api.example.com] secretName: api-tls-secret ``` Vault integration: Use Vault PKI secrets engine as issuer. Short-lived certificates (1 hour), rotated automatically. Best for internal service-to-service mTLS.

Istio: An open-source service mesh that provides traffic management, security (mTLS), and observability for microservices — without changing application code. Architecture: • Data plane: Envoy sidecar proxies injected into every pod. All inbound and outbound traffic routes through the proxy. • Control plane: Istiod (combined Pilot, Citadel, Galley). Pushes configuration to all Envoy sidecars via xDS API. Sidecar injection: • Automatic: Label namespace: istio-injection=enabled → Istiod auto-injects Envoy sidecar via MutatingWebhook • Manual: istioctl kube-inject -f deployment.yaml Key capabilities: Traffic management: • VirtualService: Fine-grained routing rules (weighted, header-based, canary) • DestinationRule: Load balancing policy, circuit breaking, mTLS settings per destination ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService spec: hosts: [reviews] http: - match: [{headers: {x-user: {exact: "beta"}}}] route: [{destination: {host: reviews, subset: v2}}] # beta users → v2 - route: - destination: {host: reviews, subset: v1} # everyone else → v1 ``` Security (mTLS): • Automatic mTLS between all services in mesh • SPIFFE-based identity: each pod gets X.509 cert tied to its ServiceAccount • AuthorizationPolicy: L7 authorization (who can call which HTTP method on which path) Observability: • Automatic metrics: Request count, latency, error rate per service pair • Distributed tracing: Automatic trace header propagation (Jaeger, Zipkin) • Access logs: Full request/response logging per proxy • Kiali: Service graph visualization (shows which services call which) Tradeoffs: Adds 5-15ms latency per hop (proxy overhead). Significant resource overhead (Envoy sidecar per pod). Operational complexity.

Gateway API: The next generation of Kubernetes traffic routing, designed to replace Ingress with a richer, more expressive, and role-oriented API. Generally Available since Kubernetes 1.28. Problems with Ingress: • Limited expressiveness: Ingress only handles basic host/path routing. Advanced routing requires implementation-specific annotations that differ per controller. • No role separation: Same person configures infrastructure and routing rules • No TCP/UDP/gRPC routing: Only HTTP(S) Gateway API roles: • Infrastructure provider: Manages GatewayClass (type of gateway — nginx, envoy, cloud LB) • Cluster operator: Creates Gateway (actual load balancer instance with listeners) • Application developer: Creates HTTPRoute (routing rules to services) Resources: ```yaml # Cluster operator: provision gateway apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: prod-gateway namespace: gateway-system spec: gatewayClassName: nginx listeners: - name: https port: 443 protocol: HTTPS tls: certificateRefs: [{name: wildcard-cert}] --- # App developer: define routing apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api-routes namespace: production spec: parentRefs: - name: prod-gateway namespace: gateway-system hostnames: [api.example.com] rules: - matches: - path: {type: PathPrefix, value: /v2} backendRefs: - name: api-v2 port: 8080 - backendRefs: - name: api-v1 port: 8080 weight: 90 - name: api-v2 port: 8080 weight: 10 # canary: 10% to v2 ``` Also supports: TCPRoute, GRPCRoute, TLSRoute for non-HTTP protocols. ReferenceGrant for cross-namespace routing.

PodDisruptionBudget: Defines the minimum number (or percentage) of pods that must be available during voluntary disruptions — node drains, rolling updates, cluster upgrades. Two types of disruptions: • Voluntary: Node maintenance (kubectl drain), rolling updates, cluster upgrades, CA scaling down nodes • Involuntary: Hardware failure, kernel panic, OOM kill — PDB doesn't protect against these PDB specification: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api-pdb spec: selector: matchLabels: app: api minAvailable: 2 # at least 2 pods must be available # OR: # maxUnavailable: 1 # at most 1 pod unavailable at a time # OR: # minAvailable: "75%" # at least 75% of replicas available ``` How it works during node drain: ```bash kubectl drain node1 --ignore-daemonsets --delete-emptydir-data ``` 1. kubectl drain marks node as unschedulable 2. For each pod on the node, drain checks if eviction would violate the PDB 3. If evicting the pod would drop below minAvailable → wait until a replacement pod is Running 4. Once replacement is healthy, evict the original pod 5. Continue with next pod Why PDB matters: • Without PDB: A node drain could evict all replicas simultaneously → service outage • With PDB minAvailable=1: At least one pod always available during drain Gotcha: PDB only works if you have enough replicas. minAvailable=2 with replicas=2 → drain is completely blocked (can never evict any pod without violating PDB). Ensure replicas > minAvailable. HPA interaction: During scale-down, HPA respects PDB. Won't reduce below minAvailable.

Both limit resource consumption but at different scopes. ResourceQuota: Limits total resource consumption within a namespace. ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: production-quota namespace: production spec: hard: requests.cpu: "20" # total CPU requests in namespace requests.memory: "40Gi" # total memory requests limits.cpu: "40" limits.memory: "80Gi" pods: "100" # max pod count persistentvolumeclaims: "20" services: "10" secrets: "50" configmaps: "50" services.loadbalancers: "5" ``` If a new pod's requests would exceed the quota → creation rejected. LimitRange: Sets default requests/limits per pod/container in a namespace. Also enforces min/max values. ```yaml apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: production spec: limits: - type: Container default: # applied when no limits specified cpu: 500m memory: 256Mi defaultRequest: # applied when no requests specified cpu: 100m memory: 128Mi max: # reject pods exceeding these cpu: "4" memory: 4Gi min: # reject pods below these cpu: 50m memory: 64Mi - type: PersistentVolumeClaim max: storage: 100Gi ``` Combined use: • LimitRange: Ensures every container has sane defaults (no resource orphans) • ResourceQuota: Caps total namespace consumption (prevents one team from monopolizing) Multi-team cluster: Each team gets their own namespace with ResourceQuota proportional to their allocation. LimitRange prevents individual containers from grabbing all quota.

Kubernetes DNS: CoreDNS runs as a Deployment in kube-system namespace. Every pod's /etc/resolv.conf points to CoreDNS — all DNS queries go through it. Service DNS names: • Same namespace: servicename (short form) • Cross-namespace: servicename.namespace • Full FQDN: servicename.namespace.svc.cluster.local Pod DNS names: • Pod IP with dashes: 10-0-0-1.namespace.pod.cluster.local • StatefulSet pods: pod-name.service-name.namespace.svc.cluster.local (stable DNS for StatefulSet) DNS resolution order (from /etc/resolv.conf in pod): ``` nameserver 10.96.0.10 # CoreDNS ClusterIP search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` For "db": tries db.default.svc.cluster.local, db.svc.cluster.local, db.cluster.local — finds it at second or third attempt. For "api.example.com": ndots:5 means fewer than 5 dots → tries search domains first, then direct query. CoreDNS configuration (Corefile): ``` .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf # forward external queries to upstream DNS cache 30 loop reload loadbalance } ``` Custom DNS: Add custom stub zones to Corefile for internal DNS (e.g., .internal resolves via corporate DNS server). Performance: CoreDNS caches with 30-second TTL. For high-query-rate services, NodeLocal DNSCache (DaemonSet) adds a local DNS cache on each node — bypasses CoreDNS for cached queries.

KEDA: A Kubernetes-based event-driven autoscaling component. Extends HPA to scale workloads based on external event sources — not just CPU/memory. Why KEDA: Standard HPA only scales on CPU/memory or custom metrics exposed within the cluster. KEDA natively integrates with external systems: • Kafka: Scale consumers based on consumer lag • RabbitMQ: Scale workers based on queue depth • AWS SQS: Scale processors based on message count • Cron: Scale up during business hours, scale to zero overnight • Prometheus: Scale based on any Prometheus query • Azure Service Bus, Redis, NATS, HTTP requests, and 50+ more scalers Scale to Zero: KEDA's key differentiator. Standard HPA minimum is 1 replica. KEDA can scale to 0 when no events and scale up from 0 when events arrive. KEDA ScaledObject: ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: order-processor-scaler spec: scaleTargetRef: name: order-processor # targets this Deployment minReplicaCount: 0 # scale to zero when idle maxReplicaCount: 50 cooldownPeriod: 300 # seconds to wait before scaling down triggers: - type: kafka metadata: bootstrapServers: kafka:9092 consumerGroup: order-processor-group topic: orders lagThreshold: "100" # scale up when lag > 100 per partition offsetResetPolicy: latest - type: cron # additional trigger: always up during business hours metadata: timezone: Asia/Kolkata start: "0 9 * * 1-5" # 9am weekdays end: "0 18 * * 1-5" # 6pm weekdays desiredReplicas: "5" ``` KEDA internals: Creates HPA objects internally. Polls external scalers to compute desired replica count. Passes metric value to HPA.

Both create Kubernetes resources, but they differ in behavior and use cases. kubectl create: • Creates a resource from scratch • Fails if the resource already exists (IdempotencyError) • Simple use: one-off resource creation • Also used for generators: kubectl create deployment web --image=nginx kubectl apply: • Creates the resource if it doesn't exist, updates it if it does • Declarative — compares desired state (YAML) with current state • Stores the applied manifest as an annotation (last-applied-configuration) for future diff computation • Idempotent — safe to run multiple times • The right choice for GitOps and CI/CD ```bash # Create: fails if deployment already exists kubectl create -f deployment.yaml # Apply: creates or updates kubectl apply -f deployment.yaml kubectl apply -f ./k8s/ # apply all files in directory kubectl apply -k ./kustomize/ # apply kustomize overlay ``` kubectl apply internals — three-way merge: 1. Last applied configuration (stored annotation) 2. Current live state (from API server) 3. New desired state (from YAML) Merge strategy: Add new fields, update changed fields, remove fields that were in last-applied but removed from YAML. kubectl diff: Preview what apply would change: ```bash kubectl diff -f deployment.yaml ``` Server-side apply (SSA): ```bash kubectl apply --server-side -f deployment.yaml ``` Diff computation happens on server — handles conflicts from multiple managers. Better for operators and GitOps tools. kubectl replace: Replaces the entire resource (not a merge). Fails if resource doesn't exist. Similar to delete + create.

A pod stays Pending when the scheduler can't find a suitable node to place it. Step 1 — Describe the pod (most informative): ```bash kubectl describe pod <pod-name> -n <namespace> ``` Look at the Events section at the bottom — the scheduler explains why it couldn't schedule. Common messages and causes: "Insufficient cpu/memory": • Pod requests more than any node has available • Fix: Check node capacity (kubectl top nodes), reduce requests, or add nodes ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOC-CPU:.status.allocatable.cpu,ALLOC-MEM:.status.allocatable.memory ``` "0/3 nodes are available: 3 Insufficient memory" — all 3 nodes lack memory: ```bash kubectl describe nodes | grep -A 10 "Allocated resources" ``` "0/3 nodes are available: 3 node(s) had untolerated taint": • Node has a taint, pod doesn't have matching toleration • Fix: Add toleration to pod, or remove taint from node "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity": • No node matches the nodeSelector or node affinity • Fix: Check node labels (kubectl get nodes --show-labels), fix affinity rules "PVC not bound" — pod waiting for persistent volume: ```bash kubectl get pvc -n <namespace> # check PVC status kubectl describe pvc <pvc-name> # see why it's not binding kubectl get sc # check StorageClass exists ``` "too many pods" — node hit maxPods limit (default 110): • Scale horizontally (add nodes) or adjust kubelet maxPods Step 2 — Check node conditions: ```bash kubectl get nodes # check Ready status kubectl describe node node1 # check Conditions and events ```

CrashLoopBackOff: The pod starts, crashes, Kubernetes restarts it, it crashes again. Kubernetes backs off the restart delay (10s → 20s → 40s → 160s → max 5min) to avoid thrashing. Step 1 — Get pod status and recent events: ```bash kubectl get pod <name> -n <namespace> kubectl describe pod <name> -n <namespace> # Check: Last State.ExitCode, Events ``` Step 2 — View container logs: ```bash kubectl logs <pod-name> -n <namespace> # current container logs kubectl logs <pod-name> -n <namespace> --previous # logs from PREVIOUS crashed instance kubectl logs <pod-name> -n <namespace> -c <container-name> # specific container ``` --previous is crucial — the current container may have just started and logs are empty. Step 3 — Decode exit code: • Exit 1: Application error — check application logs for exceptions • Exit 137: OOMKilled (memory limit exceeded) or killed externally • Exit 139: Segfault • Exit 143: SIGTERM (graceful shutdown) Check OOM: ```bash kubectl describe pod <name> | grep -i oom # "OOMKilled: true" → increase memory limit ``` Step 4 — Override command to debug: ```bash # Override command to sleep instead of crashing app kubectl run debug-pod --image=myapp:latest --command -- sleep 3600 kubectl exec -it debug-pod -- /bin/sh # Now inspect inside the container — check files, permissions, env vars ``` Common root causes: • Missing environment variable (database URL not set) • Missing config file or wrong path • Permission denied (container trying to write to read-only directory) • Wrong working directory • Java heap OOM — increase --memory limit • Health check too aggressive — pod gets killed before startup completes Step 5 — Use ephemeral debug containers: ```bash kubectl debug -it <pod-name> --image=busybox --target=<container-name> ```

Cluster Autoscaler (CA): Automatically adds nodes to the cluster when pods can't be scheduled due to insufficient resources, and removes underutilized nodes when they're no longer needed. Scale up trigger: • Scheduler marks pods Unschedulable (can't find a node that fits) • CA detects pending pods • CA calculates which node group would accommodate the pending pods • CA requests cloud provider to add a new node to the selected node group (calls EC2 Auto Scaling, GKE node pool, etc.) • New node joins, pods schedule Scale down trigger: • CA periodically checks for underutilized nodes • Node is candidate for removal if: sum of all pod requests < 50% of node's allocatable (threshold configurable) • CA checks if all pods can be rescheduled to other nodes (respects PDB, node affinity) • CA cordons (marks unschedulable), drains, and removes the node • Default check interval: 10 seconds; scale down cool-down: 10 minutes after scale up CA configuration (Helm values): ```yaml autoDiscovery: clusterName: my-cluster rbac: create: true extraArgs: scale-down-delay-after-add: 10m scale-down-unneeded-time: 10m scale-down-utilization-threshold: 0.5 max-node-provision-time: 15m skip-nodes-with-local-storage: false ``` Node groups (AWS ASG): ```yaml # Tags on ASG for auto-discovery k8s.io/cluster-autoscaler/enabled: "true" k8s.io/cluster-autoscaler/my-cluster: "owned" ``` CA limitations: • New node takes 2-5 minutes (cloud provisioning time) — KEDA + fast-start pods reduce the gap • Doesn't scale to zero node groups (use Fargate/Autopilot for serverless pods) • Can't remove nodes with local storage (hostPath, emptyDir with data) • PDB prevents draining nodes if it would violate budget Karpenter (AWS alternative): Watches pending pods directly, provisions right-sized nodes from EC2 spot/on-demand. Faster and more flexible than CA.

Init containers: Containers that run and complete BEFORE the main containers start. If an init container fails, Kubernetes restarts it until it succeeds (respecting restartPolicy). Use cases for init containers: • Run database migrations before app starts • Wait for a dependency to become available • Download configuration from a remote source • Set up shared volumes or permissions ```yaml spec: initContainers: - name: wait-for-db image: busybox command: ['sh', '-c', 'until nc -z db 5432; do sleep 2; done'] - name: run-migrations image: myapp:1.0 command: [java, -jar, app.jar, --migrate] env: - name: DB_URL value: jdbc:postgresql://db:5432/myapp containers: - name: app # starts only after all init containers succeed image: myapp:1.0 ``` Init containers vs regular containers: • Run to completion (not long-running) • Different image from main container (use minimal images for init) • Sequential: init containers run one at a time in order • Sidecar containers in Kubernetes 1.29+ are a new native concept Sidecar containers: Long-running helper containers that run alongside the main container for the pod's entire lifetime. Traditional: just put helper container in containers list. Kubernetes 1.28+ sidecar containers: ```yaml spec: initContainers: - name: log-collector # sidecar defined as init container image: fluentd restartPolicy: Always # key: this makes it a native sidecar containers: - name: app image: myapp ``` Native sidecars (K8s 1.29 stable): • Start before main containers (guaranteed ordering) • Don't block pod startup • Don't prevent pod from completing (main container exit = pod complete) • Get probes, lifecycle hooks like regular containers

GitOps: A deployment methodology where git is the single source of truth for cluster state. All changes go through git (PRs, code review). The cluster continuously reconciles to match the git state. Principles: 1. Declarative: Cluster state is described declaratively in git (YAML manifests, Helm charts, Kustomize) 2. Versioned and immutable: Git history is the audit log 3. Pulled automatically: Operators in the cluster pull from git — no push-based CI/CD with kubectl 4. Continuously reconciled: If actual state drifts from desired (manual change, crash), operator corrects it ArgoCD: The most popular GitOps operator for Kubernetes. ArgoCD Application: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: order-service namespace: argocd spec: project: production source: repoURL: https://github.com/org/k8s-configs targetRevision: main path: apps/order-service/production # Helm or Kustomize directory destination: server: https://kubernetes.default.svc namespace: production syncPolicy: automated: prune: true # delete resources removed from git selfHeal: true # revert manual changes syncOptions: - CreateNamespace=true ``` ArgoCD sync: Fetches git every 3 minutes (configurable) or via webhook. Compares git manifests vs cluster state. Applies diffs. Application health: ArgoCD checks if resources are healthy (pod Running, deployment replicas match) — not just if sync succeeded. Multi-cluster: ArgoCD can manage multiple clusters. Define cluster credentials and target each Application to a specific cluster endpoint. ApplicationSet: Auto-generates Applications — one per environment, per team, or per git directory. Powerful for managing many clusters uniformly.

Labels: Key-value pairs attached to Kubernetes objects. Used for identification and selection by other resources. ```yaml metadata: labels: app: order-service version: "1.2.3" tier: backend environment: production team: orders ``` Label conventions (Kubernetes recommended labels): ```yaml app.kubernetes.io/name: order-service app.kubernetes.io/version: "1.2.3" app.kubernetes.io/component: backend app.kubernetes.io/part-of: ecommerce-platform app.kubernetes.io/managed-by: helm ``` Selectors: Match resources by labels. Used in Services, Deployments, ReplicaSets, NetworkPolicies. ```yaml selector: matchLabels: app: order-service # exact match matchExpressions: - key: environment operator: In values: [production, staging] - key: tier operator: NotIn values: [frontend] ``` Kubectl filtering: ```bash kubectl get pods -l app=order-service kubectl get pods -l environment in (production,staging) kubectl get pods --selector="team=orders,tier=backend" ``` Annotations: Non-identifying metadata. Not used for selection. Store arbitrary information. ```yaml metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" prometheus.io/path: /actuator/prometheus kubectl.kubernetes.io/last-applied-configuration: "..." deployment.kubernetes.io/revision: "3" helm.sh/chart: order-service-1.2.3 ``` Key difference: • Labels: For Kubernetes selection and querying — must be indexed • Annotations: For tooling and documentation — not indexed, can be large (up to 256KB) • Both: Accessible via kubectl, accessible from within pods via Downward API

PriorityClass: Assigns a priority value to pods. Higher priority pods are scheduled first and, when resources are scarce, can preempt (evict) lower priority pods to make room. Creating PriorityClasses: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: critical-business value: 1000000 # higher = more priority (max 1,000,000,000) globalDefault: false preemptionPolicy: PreemptLowerPriority # or Never description: "For critical business services" --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: batch-jobs value: 100 globalDefault: false preemptionPolicy: Never # won't evict others, but gets priority in queue ``` Built-in PriorityClasses: • system-cluster-critical: 2,000,000,000 — for cluster-critical components (CoreDNS, kube-proxy) • system-node-critical: 2,000,001,000 — for node-critical components (kubelet) Assigning to pods: ```yaml spec: priorityClassName: critical-business ``` Preemption: When cluster is full and a high-priority pod is Pending, scheduler looks for lower priority pods to evict to make room. Respects PodDisruptionBudgets — won't violate PDB to preempt. Eviction order under node pressure: • BestEffort pods evicted first (no requests/limits) • Then Burstable pods (limits > requests) • Then Guaranteed pods (requests == limits) • Within same QoS class, lowest PriorityClass value first Use cases: • Guarantee critical services get scheduled in full cluster • Batch jobs have low priority — won't block real user traffic • Preemption lets time-sensitive work bump background processing

Zero-downtime deployment: Users experience no service interruption during a rolling update. Required configuration: 1. Readiness probe: New pods must pass readiness before receiving traffic. ```yaml readinessProbe: httpGet: path: /actuator/health/readiness port: 8080 initialDelaySeconds: 15 periodSeconds: 5 failureThreshold: 3 ``` 2. Rolling update strategy: ```yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # create 1 extra pod before removing old maxUnavailable: 0 # never reduce below desired count ``` 3. Graceful shutdown: App must handle SIGTERM and finish in-flight requests. ```yaml spec: terminationGracePeriodSeconds: 60 # Kubernetes waits this long ``` ```yaml lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 15"] # wait for load balancer to stop sending traffic ``` Why preStop sleep: There's a race condition — pod is removed from Service endpoints, but existing load balancer connections may still route to it for a few seconds. preStop sleep ensures the pod stops accepting new connections after it's been deregistered. 4. Minready seconds: ```yaml spec: minReadySeconds: 10 # pod must be ready for 10s before considered stable ``` 5. DB schema compatibility: During rolling update, both old and new code run simultaneously. Schema changes must be backward and forward compatible: • Add nullable columns (old code ignores them) • Don't rename or remove columns in the same deployment as code that drops them 6. Session stickiness: If using sticky sessions, ensure all sessions work with both versions during rollout (or use stateless JWT tokens). Rollout monitoring: ```bash kubectl rollout status deployment/myapp # watch until complete kubectl rollout undo deployment/myapp # instant rollback kubectl rollout history deployment/myapp # see all revisions ```

Kubernetes API: A RESTful HTTP API served by kube-apiserver. All interactions with Kubernetes (kubectl, controllers, operators, external tools) go through this API. Every resource type has CRUD endpoints. API structure: ``` GET /api/v1/pods # list all pods (core group) GET /api/v1/namespaces/prod/pods # list pods in namespace GET /api/v1/namespaces/prod/pods/web # get specific pod POST /api/v1/namespaces/prod/pods # create pod PUT /api/v1/namespaces/prod/pods/web # replace pod PATCH /api/v1/namespaces/prod/pods/web # partial update DELETE /api/v1/namespaces/prod/pods/web # Named API groups (extensions, apps, batch...) GET /apis/apps/v1/namespaces/prod/deployments GET /apis/batch/v1/namespaces/prod/jobs GET /apis/cert-manager.io/v1/certificates # CRD group ``` How kubectl works: 1. kubectl reads ~/.kube/config (kubeconfig) for server URL and credentials 2. Translates command to HTTP request: kubectl get pods → GET /api/v1/namespaces/default/pods 3. Sends HTTPS request to API server with authentication headers (certificate, token, OIDC) 4. API server authenticates, authorizes (RBAC), runs admission control 5. Reads from etcd, returns JSON response 6. kubectl renders response as human-readable table Useful kubectl flags: ```bash kubectl get pods -o json # raw JSON kubectl get pods -o yaml # YAML kubectl get pods -o wide # extra columns (node IP, etc.) kubectl get pods --watch # watch for changes kubectl -v=8 get pods # debug mode: shows HTTP requests ``` Direct API access: ```bash # Use kubectl as proxy kubectl proxy & curl http://localhost:8001/api/v1/namespaces/default/pods ```

Topology Spread Constraints: A fine-grained way to control how pods are distributed across failure domains (nodes, zones, regions). More flexible than pod anti-affinity. ```yaml spec: topologySpreadConstraints: - maxSkew: 1 # max difference between zones topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule # or ScheduleAnyway labelSelector: matchLabels: app: api - maxSkew: 1 topologyKey: kubernetes.io/hostname # also spread across nodes whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: api ``` Key fields: • maxSkew: Maximum allowed difference in pod count between any two topology domains • topologyKey: The node label key defining the topology domain • whenUnsatisfiable: - DoNotSchedule: Hard requirement (like requiredDuringScheduling) - ScheduleAnyway: Soft preference (like preferredDuringScheduling) • labelSelector: Which pods to count when computing distribution Example with 10 pods across 3 zones: • maxSkew=1: Allows distributions like 4-3-3 or 3-3-4. Disallows 5-5-0. • maxSkew=2: Allows 5-3-2 but not 6-2-2. Vs pod anti-affinity: • Anti-affinity: "Not on the same node as another app pod" — binary rule • TopologySpread: "Evenly distribute across zones, max difference of 1" — quantitative, more nuanced Default constraint (cluster-level): Cluster admins can set default topology spread constraints in scheduler config — applied to all pods that don't specify their own. Common pattern — zone spread + node spread: ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: {matchLabels: {app: api}} - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: {matchLabels: {app: api}} ```

Persistent volume binding: The process of connecting a PersistentVolumeClaim (PVC) to a matching PersistentVolume (PV). Binding process: 1. PVC is created with size, access mode, and optional StorageClass requirements 2. Kubernetes control loop finds a suitable PV (if pre-provisioned) or triggers StorageClass to dynamically provision 3. PV is bound to PVC (one-to-one relationship) 4. Pod referencing the PVC can now be scheduled on a node and mount the volume Binding criteria (PVC must match PV): • Storage capacity: PV size ≥ PVC request • Access mode: PV supports the requested mode (RWO, ROX, RWX) • StorageClass: Same storageClassName (or both empty) • VolumeMode: Block or Filesystem • Selector: PVC label selector must match PV labels Dynamic provisioning (most common in cloud): ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: order-data spec: accessModes: [ReadWriteOnce] storageClassName: fast-ssd # triggers CSI driver to create EBS/PD resources: requests: storage: 100Gi ``` StorageClass controller calls CSI driver → cloud provider creates volume → PV created automatically → bound to PVC. Binding modes: • Immediate (default): PVC bound as soon as created, before pod is scheduled • WaitForFirstConsumer: PVC bound only when a pod using it is scheduled. Enables zone-aware provisioning — volume created in same zone as pod. ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: zone-aware-ssd provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer # provision in pod's zone parameters: type: gp3 iops: "3000" throughput: "125" ``` Reclaim policy: • Retain: PV kept after PVC deletion — manual cleanup • Delete: PV (and underlying storage) deleted with PVC • Recycle: Deprecated

CSI (Container Storage Interface): A standard for exposing arbitrary storage systems to container orchestration systems. Kubernetes communicates with storage providers through the CSI API — storage vendors write CSI drivers, not Kubernetes-specific plugins. Why CSI: Before CSI, storage plugins were built into the Kubernetes codebase (in-tree plugins). Adding or updating a storage driver required a Kubernetes release. CSI separates storage from Kubernetes — drivers are deployed as pods, updated independently. CSI components deployed in cluster: • CSI driver (Node plugin): DaemonSet running on every node. Handles volume mount/unmount operations (calling OS mount commands). • CSI driver (Controller plugin): Deployment. Handles volume lifecycle: create, delete, attach, detach. • sidecar containers: external-attacher, external-provisioner, external-resizer — communicate between Kubernetes and CSI driver. CSI operations: • CreateVolume: Called when PVC is created with dynamic provisioning • DeleteVolume: Called when PVC is deleted with Delete reclaim policy • ControllerPublishVolume: Attaches volume to node (e.g., attach EBS to EC2) • NodeStageVolume: Format and mount to a global staging directory on node • NodePublishVolume: Bind mount from staging to pod's directory CSI drivers (examples): • aws-ebs-csi-driver: AWS EBS volumes — RWO, block and filesystem • aws-efs-csi-driver: AWS EFS — RWX, NFS • google-pd-csi-driver: GCP Persistent Disk • nfs-subdir-external-provisioner: NFS shares • democratic-csi: Supports TrueNAS, iSCSI, NFS CSI features: • Volume snapshots: VolumeSnapshot CRD for point-in-time snapshots • Volume cloning: Create a new PVC from an existing one • Volume expansion: Resize PVC (if StorageClass allows) • Storage capacity tracking: Scheduler knows available capacity per node

kube-proxy: A DaemonSet running on every node. Maintains network rules that implement Kubernetes Service abstractions — translating ClusterIP (virtual) to real pod IPs. What kube-proxy does: When you create a Service, kube-proxy programs the node's network to forward traffic from the Service's ClusterIP:port to one of the backing pod IPs:port. IPtables mode (default): ``` Service ClusterIP: 10.96.0.1:80 Backing pods: 10.0.1.5:8080, 10.0.2.3:8080, 10.0.3.7:8080 iptables rules (DNAT): -A KUBE-SERVICES -d 10.96.0.1 -p tcp --dport 80 \ -j KUBE-SVC-XXXX -A KUBE-SVC-XXXX -m statistic --mode random --probability 0.333 \ -j KUBE-SEP-1 (→ DNAT to 10.0.1.5:8080) -A KUBE-SVC-XXXX -m statistic --mode random --probability 0.5 \ -j KUBE-SEP-2 (→ DNAT to 10.0.2.3:8080) -A KUBE-SVC-XXXX -j KUBE-SEP-3 (→ DNAT to 10.0.3.7:8080) ``` Load balancing: random, equal-weight (not truly round-robin). Performance: O(n) rule traversal — 10,000 services = 40,000+ rules. Slow for very large clusters. IPVS mode: • Uses Linux kernel IP Virtual Server (hash table) • O(1) rule lookup — much faster at scale • Better load balancing algorithms: round-robin, least connection, source hash • Requires ipvs kernel modules ```bash kubectl get cm -n kube-system kube-proxy -o yaml | grep mode # Switch: set mode: ipvs in kube-proxy ConfigMap ``` Cilium (eBPF, replaces kube-proxy): • No iptables or IPVS at all • eBPF programs in kernel handle packet routing directly • 5-10x faster than iptables • Supports native load balancing (Maglev algorithm) • Install Cilium with kubeProxyReplacement: strict NodePort implementation: For NodePort services, kube-proxy adds rules on every node: traffic to NodeIP:NodePort → DNAT to pod IP.

Custom metrics HPA: Scale pods based on application-specific metrics (HTTP request rate, queue depth, business metrics) instead of just CPU/memory. Architecture: ``` Application → Prometheus (scrapes /metrics) → Prometheus Adapter → Kubernetes Custom Metrics API → HPA ``` Prometheus Adapter configuration: ```yaml rules: - seriesQuery: 'http_requests_total{namespace!="",pod!=""}' resources: overrides: namespace: {resource: namespace} pod: {resource: pod} name: matches: ^(.*)_total$ as: "${1}_per_second" metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m])' ``` HPA using custom metric: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 2 maxReplicas: 50 metrics: - type: Pods pods: metric: name: http_requests_per_second target: type: AverageValue averageValue: "100" # scale so each pod handles ~100 req/s - type: Resource # also scale on CPU resource: name: cpu target: type: Utilization averageUtilization: 70 ``` External metrics (from outside cluster): ```yaml - type: External external: metric: name: sqs_messages_visible selector: matchLabels: queue: orders target: type: AverageValue averageValue: "30" # scale so each pod handles ~30 messages ``` HPA behavior tuning: ```yaml behavior: scaleUp: stabilizationWindowSeconds: 30 policies: - type: Percent value: 100 periodSeconds: 15 scaleDown: stabilizationWindowSeconds: 300 # wait 5 min before scaling down policies: - type: Pods value: 2 periodSeconds: 60 ```

Managing separate configurations for dev, staging, and production requires a tool to parameterize and layer Kubernetes manifests. Kustomize (built into kubectl): ``` k8s/ base/ # common resources deployment.yaml service.yaml kustomization.yaml overlays/ dev/ kustomization.yaml # dev-specific patches resource-patch.yaml staging/ kustomization.yaml production/ kustomization.yaml replicas-patch.yaml ``` base/kustomization.yaml: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml commonLabels: app: order-service ``` overlays/production/kustomization.yaml: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: [../../base] namespace: production images: - name: myapp newTag: "1.5.0" # production-specific image tag patchesStrategicMerge: - replicas-patch.yaml configMapGenerator: - name: app-config literals: - LOG_LEVEL=WARN - DB_HOST=prod-db.internal ``` Apply: ```bash kubectl apply -k k8s/overlays/production kubectl diff -k k8s/overlays/production # preview changes ``` Helm values per environment: ```bash helm install myapp chart/ -f values.yaml -f values-production.yaml helm upgrade myapp chart/ -f values-production.yaml --set image.tag=1.5.0 ``` ArgoCD ApplicationSet for auto-generating per-environment apps: ```yaml generators: - list: elements: - env: dev cluster: dev-cluster - env: production cluster: prod-cluster template: spec: source: path: k8s/overlays/{{env}} destination: server: {{cluster}} ```

Both are Kubernetes policy engines — they enforce organizational standards and security policies by validating resources through admission webhooks. OPA Gatekeeper: • Built on Open Policy Agent (OPA) — uses Rego policy language • Installs as ValidatingWebhook + MutatingWebhook • Policy: ConstraintTemplate (defines policy schema) + Constraint (instance of policy) ```yaml # ConstraintTemplate: define the policy logic in Rego apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: k8srequiredlabels spec: crd: spec: names: {kind: K8sRequiredLabels} validation: openAPIV3Schema: properties: labels: type: array items: string targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequiredlabels violation[{"msg": msg}] { required := input.parameters.labels provided := {label | input.review.object.metadata.labels[label]} missing := required - provided count(missing) > 0 msg := sprintf("Missing required labels: %v", [missing]) } --- # Constraint: enforce the policy apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredLabels metadata: name: require-team-label spec: match: kinds: [{apiGroups: [""], kinds: [Pod]}] parameters: labels: [team, environment] ``` Kyverno (simpler, Kubernetes-native): • Policies written in YAML (no Rego) — easier to learn • Supports validation, mutation, generation, and cleanup ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-team-label spec: validationFailureAction: Enforce rules: - name: check-team-label match: any: - resources: kinds: [Pod] validate: message: "Pod must have a team label" pattern: metadata: labels: team: "?*" - name: add-default-limits # mutation rule mutate: patchStrategicMerge: spec: containers: - (name): "*" resources: limits: +(memory): 256Mi +(cpu): 500m ```

Pod preemption: When a high-priority pod is Pending because no nodes have sufficient resources, Kubernetes can evict lower-priority pods from a node to make room. Preemption process: 1. High-priority pod is created but Pending — scheduler can't find a suitable node 2. Scheduler runs preemption: looks for nodes where evicting lower-priority pods would make room 3. Scheduler selects the node that would require evicting the fewest/lowest-priority pods 4. scheduler nominates the node (sets pod.spec.nominatedNodeName) 5. Preemption controller evicts the selected pods (respecting PodDisruptionBudgets and graceful termination) 6. After termination, high-priority pod is scheduled on the now-free node Priority values: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority-batch value: 500000 preemptionPolicy: PreemptLowerPriority # can preempt --- kind: PriorityClass metadata: name: non-preempting-batch value: 100 preemptionPolicy: Never # gets priority but won't evict others ``` PriorityClass hierarchy (higher = more priority): • system-node-critical: 2,000,001,000 (kubelet, critical node agents) • system-cluster-critical: 2,000,000,000 (CoreDNS, kube-proxy) • User PriorityClasses: 0 – 1,000,000,000 • Default (no class): 0 PDB and preemption: Preemption respects PodDisruptionBudgets — won't evict a pod if it would violate the PDB. If PDB prevents eviction, preemption moves to next candidate node. Graceful eviction: Evicted pods get their full terminationGracePeriodSeconds to shut down. During this time, the high-priority pod waits. Set short grace periods for background jobs to allow fast preemption. Monitoring: prometheus metric kube_pod_status_scheduled_time shows how long pods waited for scheduling. Alert on high pending time for critical priority classes.

Canary deployment: Route a subset of traffic to the new version. Monitor metrics. Gradually increase traffic. Roll back instantly if issues arise. Approach 1 — Multiple Deployments, single Service (pod-count-based): ```yaml # stable: 9 replicas kind: Deployment metadata: name: api-stable spec: replicas: 9 template: metadata: labels: app: api version: stable # canary: 1 replica kind: Deployment metadata: name: api-canary spec: replicas: 1 template: metadata: labels: app: api version: canary # Service selects both kind: Service spec: selector: app: api # matches both stable (9) and canary (1) pods ``` 10 pods total → 10% traffic to canary (1/10 pods). Adjust ratio by changing replica counts. Approach 2 — Ingress canary (Nginx Ingress Controller): ```yaml # Canary ingress with weight annotation apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: api-canary annotations: nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "10" # 10% traffic # Or: route specific users # nginx.ingress.kubernetes.io/canary-by-header: "X-Canary" # nginx.ingress.kubernetes.io/canary-by-header-value: "always" spec: rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-canary port: {number: 8080} ``` Approach 3 — Argo Rollouts (most powerful): ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: steps: - setWeight: 5 # 5% - pause: {duration: 30m} - setWeight: 20 - pause: {} # pause indefinitely (manual approval) - setWeight: 100 analysis: # auto-promote/rollback based on metrics templates: - templateName: success-rate args: - name: service-name value: api-canary ``` Argo Rollouts integrates with Prometheus for metric-based auto-promotion and automatic rollback.

Flux: A set of open-source GitOps tools for Kubernetes (CNCF Graduated). Like ArgoCD but more modular — composed of separate controllers for different sources and reconcilers. Flux components: • source-controller: Fetches manifests from Git, Helm repos, OCI registries, S3 • kustomize-controller: Reconciles Kustomization resources (applies kustomize overlays) • helm-controller: Manages Helm releases declaratively • notification-controller: Sends alerts (Slack, PagerDuty) on reconciliation events • image-automation-controller: Automates image tag updates in git when new images are pushed Flux GitRepository: ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: k8s-configs namespace: flux-system spec: interval: 1m url: https://github.com/org/k8s-configs ref: branch: main secretRef: name: github-credentials ``` Flux Kustomization: ```yaml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: production-apps namespace: flux-system spec: interval: 10m retryInterval: 1m sourceRef: kind: GitRepository name: k8s-configs path: ./apps/production prune: true # delete resources removed from git healthChecks: - apiVersion: apps/v1 kind: Deployment name: api namespace: production ``` Flux HelmRelease: ```yaml apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: redis spec: chart: spec: chart: redis version: "19.*" sourceRef: kind: HelmRepository name: bitnami interval: 1h values: auth: enabled: false architecture: standalone ``` Flux vs ArgoCD: Flux is CLI-first, more modular, integrates better with Helm. ArgoCD has a rich UI, application grouping (ApplicationSets), RBAC for app-level access. Many teams use both: Flux for Helm chart management + ArgoCD for app visibility.

etcd is the single source of truth for Kubernetes — if it's lost, the cluster configuration is lost. Regular backups are critical. Backup with etcdctl: ```bash ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key # Verify snapshot ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table ``` Automated backup (Kubernetes CronJob): ```yaml apiVersion: batch/v1 kind: CronJob spec: schedule: "0 */6 * * *" # every 6 hours jobTemplate: spec: template: spec: hostNetwork: true # access etcd on localhost volumes: - name: etcd-certs hostPath: {path: /etc/kubernetes/pki/etcd} - name: backup hostPath: {path: /backup/etcd} containers: - name: backup image: bitnami/etcd command: - etcdctl - snapshot - save - /backup/etcd-$(date +%Y%m%d).db ``` Restore procedure: ```bash # 1. Stop kube-apiserver (prevent writes during restore) systemctl stop kube-apiserver # 2. Restore snapshot ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \ --data-dir=/var/lib/etcd-restore \ --initial-cluster=master1=https://192.168.1.1:2380 \ --initial-advertise-peer-urls=https://192.168.1.1:2380 \ --name=master1 # 3. Update etcd to use restored data directory mv /var/lib/etcd /var/lib/etcd-old mv /var/lib/etcd-restore /var/lib/etcd # 4. Restart etcd systemctl restart etcd # 5. Restart API server systemctl start kube-apiserver ``` Managed Kubernetes: EKS, GKE, AKS manage etcd for you — they handle backups and availability. For EKS, restore from a backup requires recreating the cluster. Velero: For backing up Kubernetes resource state (YAML objects) and PersistentVolume data — different from etcd backup. Velero is application-level backup; etcd is cluster-state backup.

VPA (Vertical Pod Autoscaler): Automatically adjusts CPU and memory requests and limits for containers based on actual usage patterns. Right-sizes pods without manual tuning. VPA components: • Recommender: Monitors resource usage and recommends optimal requests/limits • Admission Plugin: Applies recommendations when pods are created (mutating webhook) • Updater: Evicts pods with out-of-date resource settings so they restart with updated values VPA modes: • Off: Compute recommendations but don't apply — use kubectl to view and apply manually • Initial: Apply recommendations only on pod creation, not after • Recreate: Apply recommendations by evicting and recreating pods — causes restarts • Auto: Currently same as Recreate ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: api-vpa spec: targetRef: apiVersion: apps/v1 kind: Deployment name: api updatePolicy: updateMode: Auto resourcePolicy: containerPolicies: - containerName: api minAllowed: cpu: 100m memory: 128Mi maxAllowed: cpu: "4" memory: 8Gi controlledResources: [cpu, memory] ``` View recommendations: ```bash kubectl describe vpa api-vpa # Shows: # Recommendation: # Container Recommendations: # Container Name: api # Lower Bound: cpu: 200m, memory: 256Mi # Target: cpu: 500m, memory: 512Mi # Upper Bound: cpu: 2, memory: 2Gi ``` Limitations: • VPA and HPA cannot both manage CPU/memory on the same resource — conflict. Use VPA for resources, HPA for replicas, or use HPA with custom metrics (not CPU/memory) alongside VPA. • Pod restart required to change resource requests — causes disruption • Doesn't work well with stateful apps that can't restart freely Workflow: Run VPA in Off mode for 1 week → review recommendations → manually apply → then switch to Auto.

Multi-tenancy: Multiple teams or customers share a Kubernetes cluster with isolation between them. Soft multi-tenancy (same cluster, trust tenants): • Namespaces per tenant: Each team gets their own namespace • RBAC: Each team can only manage resources in their namespace • ResourceQuota: Cap total resources per team • NetworkPolicy: Isolate traffic between namespaces • LimitRange: Set default and max pod resources Namespace setup: ```yaml # Create namespace kubectl create namespace team-orders # RBAC: team members can manage pods/deployments in their namespace kind: RoleBinding metadata: name: team-orders-admin namespace: team-orders subjects: - kind: Group name: team-orders roleRef: kind: ClusterRole name: admin # ResourceQuota kind: ResourceQuota metadata: namespace: team-orders spec: hard: requests.cpu: "10" requests.memory: 20Gi pods: "50" # NetworkPolicy: deny all cross-namespace traffic by default kind: NetworkPolicy spec: podSelector: {} policyTypes: [Ingress, Egress] # allow intra-namespace only ingress: - from: [{podSelector: {}}] ``` Hard multi-tenancy (untrusted tenants, e.g., SaaS platform): • Separate clusters per tenant: Maximum isolation, maximum cost • Virtual clusters (vCluster): Tenant gets a virtual cluster within the physical cluster — full Kubernetes API access, isolated control plane, shared nodes • Hierarchical namespaces (HNC): Parent-child namespace relationships with policy inheritance vCluster: ```bash vcluster create tenant-a -n vcluster-tenant-a vcluster connect tenant-a -n vcluster-tenant-a # Now operating in isolated virtual cluster kubectl get nodes # shows tenant-a's virtual nodes ``` Pod Security Standards: Apply restricted profile to tenant namespaces to prevent privilege escalation across tenants.

Watch mechanism: Instead of polling the API server repeatedly, Kubernetes clients (controllers, kubelet, kubectl) watch for changes — receiving event notifications as objects change. Watch API: ```bash # HTTP long-polling watch GET /api/v1/pods?watch=true&resourceVersion=12345 # Response: stream of JSON event objects {"type": "ADDED", "object": {...pod...}} {"type": "MODIFIED", "object": {...pod...}} {"type": "DELETED", "object": {...pod...}} ``` ResourceVersion: Each object has a resourceVersion. Watch from a specific resourceVersion to receive only events after that point. Informers (client-go): Informers are the standard abstraction for watching in Kubernetes controllers. They combine: 1. List: Initial list of all existing objects 2. Watch: Stream of change events from the point of the list 3. Local cache: In-memory store of all objects (from list + watch events) 4. Re-sync: Periodically re-lists to catch any missed events 5. Event handlers: Callbacks for Add/Update/Delete events ```go // Controller using informers (simplified) podInformer := informerFactory.Core().V1().Pods() podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*corev1.Pod) // put in work queue controller.queue.Add(pod.Namespace + "/" + pod.Name) }, UpdateFunc: func(old, new interface{}) { ... }, DeleteFunc: func(obj interface{}) { ... }, }) // Worker reads from queue, reconciles func (c *Controller) processNextItem() bool { key, quit := c.queue.Get() pod, err := c.podLister.Get(key) // from local cache, not API server c.reconcile(pod) } ``` Work queue: Event handlers don't process directly — they enqueue keys. Workers dequeue and reconcile. Automatic retry on failure. De-duplication: multiple changes to the same object result in one reconciliation. SharedInformerFactory: Multiple controllers watching the same resource share one informer — one list+watch per resource type regardless of how many controllers need it.

eBPF (extended Berkeley Packet Filter): A Linux kernel technology that allows running sandboxed programs in the kernel without changing kernel source code or loading kernel modules. eBPF programs are attached to kernel hooks — system calls, network events, tracepoints — and execute at kernel speed. Why eBPF for Kubernetes networking: Traditional Kubernetes networking uses iptables (managed by kube-proxy). iptables has O(n) rule traversal — 10,000 services = 40,000+ iptables rules, significant latency. eBPF replaces iptables with hash-table-based routing in the kernel — O(1) lookup, 2-5x less overhead. Cilium: The leading eBPF-based CNI plugin for Kubernetes. Cilium capabilities: 1. kube-proxy replacement: ```bash # Cilium replaces kube-proxy entirely helm install cilium cilium/cilium \ --set kubeProxyReplacement=strict \ --set k8sServiceHost=api.mycluster.com ``` All Service traffic handled by eBPF maps instead of iptables. 2. Identity-based security: Pod identity based on labels, not IPs. IP addresses are ephemeral — Cilium uses cryptographic identities. NetworkPolicy enforcement at L3/L4 and L7 (HTTP path, method, gRPC service). 3. Hubble — network observability: ```bash hubble observe --namespace production --type trace # Shows: source pod → destination pod, protocol, verdict (FORWARDED/DROPPED), http status ``` Real-time network flow visibility without any application changes. 4. Transparent encryption: IPsec or WireGuard encryption for all pod-to-pod traffic across nodes. No application changes. 5. Bandwidth management: Rate limiting per pod at the network level — eBPF Token Bucket algorithm. 6. Load balancing: Maglev hash-based load balancing (Google's algorithm) — consistent hashing for minimal connection disruption during backend changes.

mTLS (mutual TLS): Both client and server present certificates — proves identity in both directions. Ensures services can only be called by authenticated, authorized callers — even within the cluster. Approach 1 — Istio (service mesh, no code change): ```yaml # Enable mTLS cluster-wide apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system # applies to entire mesh spec: mtls: mode: STRICT # reject non-mTLS connections ``` Istio identity: Each service account gets an X.509 certificate (SPIFFE format: spiffe://cluster.local/ns/production/sa/order-service). Certificates issued by Istiod CA, rotated every 24 hours. Authorization with mTLS identity: ```yaml apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: order-service-authz namespace: production spec: selector: matchLabels: app: order-service action: ALLOW rules: - from: - source: principals: - cluster.local/ns/production/sa/api-gateway # only api-gateway can call - to: - operation: methods: [GET, POST] paths: [/api/orders*] ``` Approach 2 — SPIFFE/SPIRE (identity framework): SPIFFE: Standard for service identity. SPIRE: Implementation that issues SPIFFE X.509 SVIDs (Service Verifiable Identity Documents). Works with any mTLS-capable service — not mesh-dependent. Approach 3 — cert-manager with manual configuration: Provide each service with a TLS keypair. Application code loads certificate, presents in TLS handshake. More work but no service mesh overhead. Approach 4 — Cilium mTLS: Cilium can enforce mTLS between pods using WireGuard encryption at the network layer — no application involvement, no sidecar proxy.

Velero: An open-source tool for backing up Kubernetes cluster resources and persistent volumes. Unlike etcd backups (cluster state), Velero backs up at the application level — namespaces, deployments, services, ConfigMaps, PVCs, and the actual PV data. What Velero backs up: • Kubernetes object definitions (JSON from API server) • Persistent Volume data (via volume snapshots or Restic/Kopia file-level backup) Installation: ```bash # Install Velero with AWS S3 backend velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.9.0 \ --bucket my-velero-backups \ --backup-location-config region=us-east-1 \ --snapshot-location-config region=us-east-1 \ --secret-file ./aws-credentials ``` Create backup: ```bash # Backup entire cluster velero backup create full-cluster-backup # Backup specific namespace velero backup create production-backup --include-namespaces production # Backup with PV snapshot velero backup create production-backup \ --include-namespaces production \ --snapshot-volumes # Scheduled backup velero schedule create daily-backup \ --schedule="0 2 * * *" \ --include-namespaces production \ --ttl 720h # 30-day retention ``` Restore: ```bash velero restore create --from-backup production-backup velero restore create --from-backup production-backup --include-namespaces production velero restore describe my-restore # check restore status ``` Disaster recovery scenario: 1. Cluster is destroyed 2. Provision new cluster 3. Install Velero and point to S3 backup location 4. Velero syncs backup metadata from S3 5. Run velero restore to recreate all resources and PV data Limitations: • Not a substitute for application-level backups (DB dump) — PV snapshot may be crash-consistent but not application-consistent • Restore to a different cloud provider requires export of PV data (Kopia/Restic instead of snapshots)

Cluster upgrades must be done carefully — components must stay within supported version skew, and upgrades must not interrupt running workloads. Version skew policy: • kube-apiserver: Must be upgraded first • kubelet: Can be 1-2 minor versions behind API server • kubectl: Can be ±1 minor version from API server • Always upgrade one minor version at a time (1.27 → 1.28 → 1.29, not 1.27 → 1.29) Managed Kubernetes (EKS, GKE, AKS): ```bash # EKS upgrade eksctl upgrade cluster --name my-cluster --version 1.29 --approve # GKE upgrade (auto or manual) gcloud container clusters upgrade my-cluster --master --cluster-version 1.29 gcloud container clusters upgrade my-cluster --node-pool default-pool ``` Self-managed cluster (kubeadm): ```bash # 1. Upgrade control plane kubeadm upgrade plan # check available versions kubeadm upgrade apply v1.29.0 # 2. Upgrade kubectl and kubelet on control plane apt-get install -y kubelet=1.29.0-00 kubectl=1.29.0-00 systemctl restart kubelet # 3. Upgrade worker nodes (one at a time) kubectl cordon node-1 kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data # On node-1: kubeadm upgrade node apt-get install -y kubelet=1.29.0-00 systemctl restart kubelet # Back on control plane: kubectl uncordon node-1 ``` Pre-upgrade checklist: • Review API deprecations (deprecated APIs removed in new version) • Check add-on compatibility: Helm charts, CNI, CSI, admission controllers • Test upgrade in staging cluster first • Verify all PodDisruptionBudgets allow node drains • Backup etcd • Ensure enough spare capacity for rolling node upgrades API deprecation handling: ```bash # Find deprecated API usage kubectl deprecations # requires pluto tool or similar pluto detect-all-in-cluster ```

Workload identity: Pods get cloud IAM identities without long-lived credential management. Instead of mounting AWS/GCP credentials in pods, the pod's Kubernetes ServiceAccount maps to a cloud IAM role. Authentication happens via short-lived OIDC tokens. IRSA (IAM Roles for Service Accounts) — AWS EKS: How it works: 1. EKS exposes an OIDC endpoint (https://oidc.eks.us-east-1.amazonaws.com/id/...) 2. Kubernetes mounts a projected ServiceAccount token into the pod (audience: sts.amazonaws.com) 3. AWS SDK inside the pod gets the token, calls AWS STS AssumeRoleWithWebIdentity 4. STS verifies the token with the EKS OIDC endpoint 5. STS returns temporary credentials for the mapped IAM role 6. Pod accesses AWS services with the role's permissions Setup: ```bash # 1. Create IAM OIDC provider for cluster eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve # 2. Create IAM role with trust policy export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export OIDC_PROVIDER=$(aws eks describe-cluster --name my-cluster --query cluster.identity.oidc.issuer --output text | sed -e "s/^https:\/\///") cat > trust-policy.json <<EOF { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"}, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": {"StringEquals": {"${OIDC_PROVIDER}:sub": "system:serviceaccount:production:order-service-sa"}} }] } EOF # 3. Annotate ServiceAccount kubectl annotate sa order-service-sa -n production \ eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/OrderServiceRole ``` GKE Workload Identity: Same concept, uses GCP OIDC. Maps Kubernetes SA to GCP Service Account. Annotate with iam.gke.io/gcp-service-account. Azure Workload Identity: Uses Azure AD Federated Identity Credentials.

Prometheus Operator: Manages Prometheus instances in Kubernetes using CRDs (ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager). Eliminates manual Prometheus configuration. Installation (kube-prometheus-stack Helm chart — includes everything): ```bash helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ -n monitoring --create-namespace \ -f values.yaml ``` Includes: Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter. ServiceMonitor — auto-discover metrics: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: order-service namespace: monitoring labels: release: kube-prometheus-stack # must match Prometheus selector spec: namespaceSelector: matchNames: [production] selector: matchLabels: app: order-service # selects the Service with this label endpoints: - port: http-metrics path: /actuator/prometheus interval: 15s ``` PrometheusRule — define alerts: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: order-service-alerts namespace: monitoring spec: groups: - name: order-service.rules rules: - alert: HighErrorRate expr: | rate(http_server_requests_seconds_count{status=~"5..",app="order-service"}[5m]) / rate(http_server_requests_seconds_count{app="order-service"}[5m]) > 0.05 for: 5m labels: severity: warning annotations: summary: "High error rate on {{ $labels.app }}" ``` Grafana dashboards: Import by ID (4701 for Spring Boot JVM, 7249 for Kubernetes cluster). Custom dashboards via ConfigMaps with grafana.com/dashboard annotation. Alertmanager routing: ```yaml route: receiver: slack-warnings routes: - match: severity: critical receiver: pagerduty ```

External Secrets Operator (ESO): A Kubernetes operator that synchronizes secrets from external secret management systems (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault) into Kubernetes Secrets. Why ESO: • Kubernetes Secrets are only base64-encoded by default (not secure) • Secrets stored in git (even encrypted) create complexity • Secret rotation: ESO auto-syncs when external secret changes • Single source of truth: All secrets in one place (Vault/AWS), Kubernetes is a consumer Concepts: • SecretStore / ClusterSecretStore: Defines where to fetch secrets from (provider + auth) • ExternalSecret: Specifies which external secret to sync and what Kubernetes Secret to create AWS Secrets Manager setup: ```yaml apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: aws-secrets-manager spec: provider: aws: service: SecretsManager region: us-east-1 auth: jwt: # IRSA-based auth serviceAccountRef: name: external-secrets-sa namespace: external-secrets --- apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials namespace: production spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials # creates this Kubernetes Secret creationPolicy: Owner data: - secretKey: DB_PASSWORD # key in Kubernetes Secret remoteRef: key: prod/database/credentials # AWS secret name property: password # JSON property in the secret - secretKey: DB_USERNAME remoteRef: key: prod/database/credentials property: username ``` Secret rotation: When AWS secret is rotated, ESO re-syncs to Kubernetes Secret within refreshInterval. For pods to pick up changes: either use volume mounts (auto-refreshed) or implement secret reload mechanism (SIGHUP or restarting pods). HashiCorp Vault integration: Same pattern but uses VaultAuth resource. Supports dynamic secrets (Vault generates short-lived DB credentials on demand).

Kubernetes networking issues: pods can't reach each other, service discovery failing, external traffic not reaching pods. Step 1 — Determine the scope: ```bash # Pod-to-pod on same node? # Pod-to-pod across nodes? # Pod-to-service? # External-to-service (Ingress/LoadBalancer)? ``` Step 2 — Test pod-to-pod connectivity: ```bash # From source pod, try to reach destination pod directly kubectl exec -it source-pod -- curl -v http://10.0.1.5:8080/health kubectl exec -it source-pod -- ping 10.0.1.5 kubectl exec -it source-pod -- nc -zv 10.0.1.5 8080 ``` Step 3 — Test service DNS resolution: ```bash kubectl exec -it source-pod -- nslookup order-service kubectl exec -it source-pod -- nslookup order-service.production.svc.cluster.local kubectl exec -it source-pod -- curl order-service:8080/health ``` Step 4 — Check service endpoints: ```bash kubectl get endpoints order-service # should show pod IPs # If empty: service selector doesn't match pod labels kubectl get pods --show-labels | grep order # verify labels kubectl describe service order-service # check selector ``` Step 5 — Network policy check: ```bash kubectl get networkpolicies -n production # list policies # Simulate: does policy allow source pod to reach destination? # Use Cilium Hubble or Calico policy viewer ``` Step 6 — Node-level networking: ```bash # Check iptables rules (kube-proxy mode) iptables -t nat -L KUBE-SERVICES | grep 10.96.0.1 # find service IP rules # Check overlay (Calico) kubectl exec -it -n kube-system $(kubectl get pod -n kube-system -l k8s-app=calico-node -o name | head -1) -- calicoctl node status ``` Step 7 — DNS troubleshooting: ```bash kubectl exec -it source-pod -- cat /etc/resolv.conf # check DNS config kubectl logs -n kube-system -l k8s-app=kube-dns # CoreDNS logs kubectl exec -it source-pod -- nslookup kubernetes # test CoreDNS basic ``` Tools: netshoot (nicolaka/netshoot) — a debugging container with all networking tools. kubectl debug ephemeral containers for distroless pods.

Node pressure: When a node runs low on resources (memory, disk, PIDs), kubelet starts evicting pods to free resources and prevent node failure. Eviction signals: • memory.available: Available node memory • nodefs.available: Filesystem space for node • nodefs.inodesFree: Filesystem inodes • imagefs.available: Filesystem for container images • pid.available: Available process IDs Eviction thresholds (kubelet config): ```yaml evictionHard: memory.available: "100Mi" # hard eviction when < 100Mi free nodefs.available: "10%" imagefs.available: "15%" evictionSoft: memory.available: "500Mi" # soft eviction evictionSoftGracePeriod: memory.available: "1m30s" # wait this long before soft eviction evictionMinimumReclaim: memory.available: "200Mi" # reclaim at least 200Mi per eviction ``` Eviction process: 1. kubelet detects threshold crossed 2. Selects pods to evict (lowest priority first, within priority: most resource-consuming first) 3. Sends SIGTERM, waits terminationGracePeriodSeconds 4. If hard eviction: immediate SIGKILL 5. Resources freed, node pressure condition clears Eviction order: 1. BestEffort pods (no requests/limits) 2. Burstable pods exceeding their requests 3. Guaranteed pods (last resort) Within same QoS class: pods using most resources relative to requests evicted first. Node conditions: ```bash kubectl describe node node1 | grep -A 10 Conditions # MemoryPressure: True → kubelet is evicting pods # DiskPressure: True → disk eviction happening # PIDPressure: True → too many processes ``` Preventing eviction: • Set resource limits: Guaranteed QoS pods are last to be evicted • PriorityClass: High-priority pods evicted last within QoS class • Node over-provisioning: Keep buffer capacity on nodes • Cluster Autoscaler: Adds nodes before pressure occurs

Ephemeral containers: Temporary containers added to a running pod for debugging. Unlike regular containers, they can be added after pod creation, don't have resource limits that affect QoS class, and aren't restarted if they exit. Why needed: Distroless or minimal images have no debugging tools (no shell, no curl, no ps). You can't modify a running pod's containers. Ephemeral containers let you inject debug tools without rebuilding the image or restarting the pod. Adding an ephemeral container: ```bash # Add a debug container to a running pod kubectl debug -it my-pod \ --image=gcr.io/distroless/java21-debug \ --target=myapp # share process namespace with target container # Or use a generic debug image with all tools kubectl debug -it my-pod \ --image=nicolaka/netshoot \ --target=myapp ``` Within the ephemeral container: ```bash # Share PID namespace with target container (--target) ps aux # see target container processes # Attach to target container's filesystem via /proc ls /proc/1/root/app/ # target container's filesystem # Network debugging ss -tlnp # see target container's ports curl localhost:8080/actuator/health ``` Copy-and-debug (node-level debugging): ```bash # Create a copy of the pod with a debug container and changed entrypoint kubectl debug my-pod --copy-to=debug-pod --share-processes \ --image=busybox --set-image=myapp=busybox # replace myapp with debug image ``` Node debugging: ```bash # Debug a specific node (creates privileged pod) kubectl debug node/my-node -it --image=ubuntu # Node filesystem mounted at /host chroot /host # access node filesystem ``` Ephemeral container limitations: • Can't add ports or volume mounts • No probes • Process namespace sharing must be enabled on the pod (shareProcessNamespace: true) • Target container process namespace access requires the --target flag

Zero-trust networking: Default deny everything. Explicitly allow only required connections. Every connection is authenticated and authorized — no implicit trust based on network location. Step 1 — Default deny all in every namespace: ```yaml # Apply to each namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} # all pods policyTypes: - Ingress - Egress # deny all ingress AND egress ``` Step 2 — Allow DNS (required for all pods): ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns namespace: production spec: podSelector: {} policyTypes: [Egress] egress: - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP ``` Step 3 — Service-specific policies: ```yaml # Order Service: allow ingress from API Gateway only kind: NetworkPolicy metadata: name: order-service-policy namespace: production spec: podSelector: matchLabels: app: order-service policyTypes: [Ingress, Egress] ingress: - from: - podSelector: matchLabels: app: api-gateway ports: [{port: 8080}] egress: - to: - podSelector: matchLabels: app: postgres ports: [{port: 5432}] - to: - podSelector: matchLabels: app: kafka ports: [{port: 9092}] # Allow DNS (covered by default DNS policy above) ``` Cilium L7 policies (HTTP-level): ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy spec: endpointSelector: matchLabels: app: order-service ingress: - fromEndpoints: - matchLabels: app: api-gateway toPorts: - ports: [{port: "8080"}] rules: http: - method: GET path: /api/orders.* # only allow GET /api/orders* ```

Scheduler Framework: A pluggable architecture that allows custom plugins to be integrated into the kube-scheduler without modifying core code. Each phase of scheduling can be extended. Scheduling cycle phases: 1. QueueSort: Sort pods in the scheduling queue (default: priority-based) 2. PreFilter: Check preconditions and compute state for other plugins 3. Filter: Eliminate nodes that can't run the pod (runs all filter plugins) 4. PostFilter: Called if all nodes were filtered out — can preempt to make room 5. PreScore: Prepare state for scoring 6. Score: Rank remaining nodes (each plugin returns 0-100) 7. NormalizeScore: Normalize scores across plugins 8. Reserve: Reserve resources on selected node (prevent race conditions) 9. Permit: Approve/deny/wait the binding decision 10. PreBind: Preparation before binding (e.g., provision PVCs) 11. Bind: Commit the pod to the node (set pod.spec.nodeName) 12. PostBind: Cleanup after successful bind Custom scheduler plugin (Go): ```go type GPUTypePlugin struct{} func (p *GPUTypePlugin) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status { requiredGPU := pod.Annotations["gpu-type"] if requiredGPU == "" { return nil // no requirement, pass } nodeGPU := nodeInfo.Node().Labels["gpu-type"] if nodeGPU != requiredGPU { return framework.NewStatus(framework.Unschedulable, fmt.Sprintf("node GPU type %s != required %s", nodeGPU, requiredGPU)) } return nil } ``` Register and deploy as a custom scheduler: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: my-custom-scheduler plugins: filter: enabled: [{name: GPUTypePlugin}] ``` Deploying custom scheduler as a second scheduler: Pods can select which scheduler to use via pod.spec.schedulerName. Some pods use default-scheduler, specialized pods use my-custom-scheduler.

Secret rotation: Automatically updating secrets (DB passwords, API keys) before they expire or after a potential compromise — without downtime. Challenge: When a Kubernetes Secret is updated, pods with env-var injection don't see the change until restarted. Volume-mounted secrets propagate automatically (within ~1 minute). Approach 1 — External Secrets Operator + volume mounts: ```yaml # ExternalSecret auto-refreshes from Vault/AWS SM spec: refreshInterval: 1h target: name: db-credentials --- # Pod: volume mount (auto-refreshes when Secret changes) volumes: - name: db-creds secret: secretName: db-credentials containers: - volumeMounts: - name: db-creds mountPath: /secrets readOnly: true # App reads /secrets/password at each DB connection ``` Approach 2 — Reloader (reload deployment on secret change): ```yaml # Stakater Reloader — watches secrets and restarts pods annotations: secret.reloader.stakater.com/reload: "db-credentials" ``` When the Secret updates, Reloader performs a rolling restart of the Deployment. Approach 3 — Vault Agent Injector (dynamic secrets): ```yaml annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-secret-db: "database/creds/my-role" vault.hashicorp.com/role: "order-service" vault.hashicorp.com/agent-inject-template-db: | {{- with secret "database/creds/my-role" -}} DB_PASSWORD={{ .Data.data.password }} {{- end }} ``` Vault Agent sidecar fetches short-lived credentials (e.g., 1 hour TTL). Agent renews before expiry. App reads from file. When credentials change, Agent rewrites the file. Approach 4 — Application handles rotation: ```java // App periodically re-reads credentials file ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1); scheduler.scheduleAtFixedRate(() -> { String newPassword = Files.readString(Path.of("/secrets/password")); dbPool.updateCredentials(newPassword); }, 0, 5, TimeUnit.MINUTES); ```

API Aggregation Layer: A mechanism to extend the Kubernetes API server with additional APIs served by separate API servers (aggregated servers). These appear seamlessly under the same kube-apiserver endpoint. How it works: 1. Register an APIService object pointing to a backend service 2. kube-apiserver proxies requests for that API group to the backend 3. Clients call kube-apiserver normally — transparent routing APIService: ```yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io version: v1beta1 service: name: metrics-server namespace: kube-system port: 443 groupPriorityMinimum: 100 versionPriority: 100 caBundle: <base64-CA> # CA for the backend service's TLS insecureSkipTLSVerify: false ``` metrics-server uses API aggregation: kubectl top pods works because metrics-server registers metrics.k8s.io/v1beta1. When kubectl top calls /apis/metrics.k8s.io/v1beta1/pods, kube-apiserver proxies to metrics-server. CRD vs API Aggregation: • CRD: Simple, managed by API server, no separate process. Limited customization (no custom storage, no server-side logic beyond admission). • API Aggregation: Full custom server. Custom storage backends, custom business logic, custom validation. More complex — separate deployment. When to use API Aggregation: • Custom storage backend (not etcd) • Streaming responses (WebSocket, server-sent events) • Fine-grained control over API behavior • Migrating from external to in-cluster APIs Examples using aggregation: metrics-server, Kubernetes Service Catalog, custom cloud provider APIs, API gateways built into Kubernetes.

Kubernetes API server is the cluster's control interface — securing it is foundational. Authentication (who are you): • X.509 client certificates: kubeconfig has client cert+key. API server verifies cert signed by cluster CA. Used by kubeadm admin, service accounts (historical). • ServiceAccount tokens: Pods authenticate with auto-mounted tokens (JWT). Kubernetes verifies token signature. • OIDC: Integrate with identity providers (Okta, Dex, Google IAM). Human users log in via SSO, get JWT, kubectl uses it. Best for team access. • Bootstrap tokens: Short-lived tokens for node joining. Authorization (what can you do): • RBAC: Role-based access control on API resources • Node: kubelet can only access resources related to its own node • ABAC: Attribute-based (legacy, avoid) • Webhook: External authorization service Securing the API server: ```bash # API server flags (kubeadm cluster) --anonymous-auth=false # disable anonymous access --authorization-mode=Node,RBAC # enable RBAC --enable-admission-plugins=NodeRestriction,PodSecurity,... --audit-log-path=/var/log/kubernetes/audit.log --audit-log-maxage=30 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --tls-min-version=VersionTLS12 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,... ``` Network security: • API server typically on port 6443 — restrict access at network level • In cloud: Use security groups/firewall rules to allow only CI/CD systems and admin bastion • Private endpoint: EKS/GKE support private cluster — API server only accessible from VPC Audit logging: ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: RequestResponse resources: - group: "" resources: [secrets] # log all secret access - level: Metadata omitStages: [RequestReceived] resources: [pods, deployments] - level: None users: [system:kube-proxy] # don't log noisy system requests ``` Kubeconfig rotation: Use short-lived OIDC tokens instead of static admin certificates. Rotate certificates annually at minimum.

Headless service: A service with clusterIP: None. Instead of returning a single virtual ClusterIP, DNS queries return the IPs of all backing pods directly. Regular service DNS: kubernetes-service.namespace.svc.cluster.local → single ClusterIP (load-balanced virtual IP). Headless service DNS: headless-service.namespace.svc.cluster.local → multiple A records (one per pod IP). Client receives all pod IPs and chooses. ```yaml apiVersion: v1 kind: Service metadata: name: kafka spec: clusterIP: None # headless selector: app: kafka ports: - port: 9092 ``` DNS response: ```bash nslookup kafka # Returns: # kafka → 10.0.1.5 # kafka → 10.0.2.3 # kafka → 10.0.3.7 ``` StatefulSet DNS: Headless service used with StatefulSets gives each pod a stable DNS name: ``` pod-0.kafka.namespace.svc.cluster.local → 10.0.1.5 (kafka-0 pod) pod-1.kafka.namespace.svc.cluster.local → 10.0.2.3 (kafka-1 pod) ``` When to use headless services: 1. StatefulSets: Required for stable pod DNS (databases, Kafka, ZooKeeper). Each pod addressed individually for replication, leader election. 2. gRPC client-side load balancing: gRPC uses persistent HTTP/2 connections — a single ClusterIP always routes to the same backend (connection-level, not request-level). Headless service gives gRPC client all pod IPs for true request-level load balancing. 3. Client-side load balancing: When the application implements its own load balancing (Kafka client knows all brokers, Cassandra driver knows all nodes). 4. Direct pod access: When you need to connect to a specific pod, not just any pod behind the service. ExternalName service: Another ClusterIP: none variant that maps to an external DNS name (not pod-based).

Both are admission webhook types that plug into the Kubernetes API server's admission control pipeline. They intercept API requests before resources are persisted. MutatingWebhookConfiguration: • Called first (before validating webhooks) • Can MODIFY the resource (add/change fields) • Multiple mutating webhooks can run, each modifying the object • Common uses: inject sidecar containers, set default values, add labels/annotations, normalize user input ValidatingWebhookConfiguration: • Called after mutating webhooks • Can only APPROVE or REJECT — cannot modify • Can run in parallel (multiple validating webhooks for same resource) • Common uses: policy enforcement, constraint validation, security checks Webhook request flow: ``` kubectl apply deployment.yaml → API Server → Authentication + Authorization → MutatingWebhook1 (injects sidecar) → MutatingWebhook2 (adds labels) → Object schema validation → ValidatingWebhook1 (check required labels) → ValidatingWebhook2 (check image from approved registry) → Persist to etcd ``` Webhook implementation (Spring Boot example): ```java @PostMapping("/validate/pods") public AdmissionReview validate(@RequestBody AdmissionReview review) { Pod pod = (Pod) review.getRequest().getObject(); String image = pod.getSpec().getContainers().get(0).getImage(); boolean allowed = image.startsWith("myregistry.company.com/"); String message = allowed ? null : "Images must be from company registry"; return AdmissionReview.builder() .response(AdmissionResponse.builder() .uid(review.getRequest().getUid()) .allowed(allowed) .status(allowed ? null : Status.builder().message(message).build()) .build()) .build(); } ``` Failure policy: • Fail: Reject the request if webhook is unreachable (safe, but may block cluster operations) • Ignore: Allow the request if webhook is unreachable (less safe, allows policy bypass if webhook is down) Timeout: Webhooks must respond within timeoutSeconds (default 10s). Slow webhooks block API operations.

Multi-environment GitOps: Changes flow through environments (dev → staging → production) via git — no direct kubectl access to any environment. Repository structure: ``` k8s-configs/ base/ order-service/ deployment.yaml service.yaml kustomization.yaml overlays/ dev/ kustomization.yaml # base + dev-specific patches staging/ kustomization.yaml production/ kustomization.yaml ``` Promotion workflow: 1. Developer merges feature branch to main 2. CI/CD (GitHub Actions) builds new image, tags it: myapp:sha-abc123 3. CI/CD opens a PR to k8s-configs: updates overlays/dev/kustomization.yaml with new image tag 4. Auto-merge to k8s-configs dev branch 5. ArgoCD syncs dev environment → new image deployed to dev 6. After testing, open PR to update staging overlay 7. Merge → ArgoCD syncs staging 8. After staging validation, open PR to update production overlay 9. Require manual approval on production PR (branch protection rule) 10. Merge → ArgoCD syncs production Automating image tag updates: ```bash # In CI/CD after pushing image: git clone k8s-configs cd k8s-configs kustomize edit set image myapp=myregistry/myapp:sha-${GITHUB_SHA} \ --overlay overlays/dev git commit -am "deploy: myapp sha-${GITHUB_SHA} to dev" git push # Open PR for staging via gh pr create... ``` Flux image automation: ```yaml apiVersion: image.toolkit.fluxcd.io/v1beta2 kind: ImageUpdateAutomation spec: git: checkout: ref: {branch: main} commit: author: {email: flux@company.com, name: Flux} push: {branch: main} update: path: ./overlays/dev strategy: Setters ``` Flux can automatically update image tags in git when new images are pushed to registry.

Cluster logging: Collect logs from all pods across all nodes, aggregate them centrally, make them searchable and alertable. Logging architecture options: 1. DaemonSet-based (most common — Fluent Bit): Fluent Bit runs on every node, reads pod logs from /var/log/containers/, parses and ships to central backend. ```yaml # Fluent Bit DaemonSet (simplified) apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: logging spec: selector: matchLabels: {k8s-app: fluent-bit} template: spec: volumes: - name: varlog hostPath: {path: /var/log} - name: containers-log hostPath: {path: /var/lib/docker/containers} containers: - name: fluent-bit image: fluent/fluent-bit:2.1 volumeMounts: - name: varlog mountPath: /var/log - name: containers-log mountPath: /var/lib/docker/containers readOnly: true ``` Fluent Bit configuration: ```ini [INPUT] Name tail Path /var/log/containers/*.log Parser cri Tag kube.* [FILTER] Name kubernetes Match kube.* Merge_Log On Keep_Log Off K8S-Logging.Parser On [OUTPUT] Name es Match * Host elasticsearch Port 9200 Index kubernetes ``` 2. Loki + Grafana (lightweight): ```yaml # Promtail DaemonSet ships to Loki # Grafana queries Loki with LogQL logql_query: | {namespace="production", pod=~"order-service-.*"} |= "ERROR" | json | line_format "{{.time}} {{.level}} {{.message}}" ``` 3. Cloud-native: • AWS: CloudWatch Logs via aws-for-fluent-bit • GCP: Cloud Logging auto-collects GKE pod logs Log format best practices: • JSON structured logs with consistent fields: timestamp, level, service, traceId, spanId • Index on namespace, pod name, log level for efficient querying

Projected ServiceAccount Token: A Kubernetes-generated JWT with configurable audience, expiration, and binding to specific objects. More secure than the old auto-mounted secret-based tokens. Old approach (before K8s 1.22): Kubernetes automatically created a Secret with a permanent ServiceAccount token. The secret never expired and could be reused across clusters. New approach (Bound ServiceAccount Tokens, K8s 1.20+, default since 1.24): • Tokens are time-limited (default 1 hour) • Bound to specific pod (pod UID embedded) — if pod is deleted, token is invalidated • Bound to specific service account • Can specify audience (e.g., sts.amazonaws.com for IRSA) Projected volume configuration: ```yaml spec: volumes: - name: token-vol projected: sources: - serviceAccountToken: path: token expirationSeconds: 3600 # 1 hour audience: my-service # restrict token audience - configMap: name: kube-root-ca.crt items: - key: ca.crt path: ca.crt - downwardAPI: items: - path: namespace fieldRef: fieldPath: metadata.namespace containers: - volumeMounts: - name: token-vol mountPath: /var/run/secrets/kubernetes.io/serviceaccount readOnly: true ``` Token refresh: kubelet automatically refreshes the token before it expires (at 80% of expiration). The application reads the token file — it always gets a current token if it re-reads before each use. IRSA token: EKS uses projected tokens with audience=sts.amazonaws.com. AWS STS exchanges this token for IAM role credentials via OIDC federation. Validating projected tokens: ```bash # Decode the token kubectl exec my-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | \ cut -d. -f2 | base64 -d 2>/dev/null | jq # Shows: iss, sub (system:serviceaccount:ns:sa), aud, exp, iat, pod identity ```

Database connection pooling in Kubernetes: Each pod has its own connection pool. With many replicas, total connections to the database can overwhelm it. Problem: 100 pods × 10 connections each = 1,000 connections. PostgreSQL default max_connections = 100. Not scalable. Solution 1 — PgBouncer (connection pooler as sidecar): ```yaml spec: containers: - name: app image: myapp env: - name: DB_HOST value: localhost:5432 # connect to pgbouncer on localhost - name: pgbouncer image: pgbouncer/pgbouncer env: - name: DB_HOST value: postgres.production.svc - name: POOL_SIZE value: "10" - name: POOL_MODE value: transaction # transaction pooling ``` Solution 2 — PgBouncer as a shared Kubernetes Service: ```yaml # One PgBouncer deployment for all apps kind: Deployment metadata: name: pgbouncer spec: replicas: 3 template: spec: containers: - name: pgbouncer image: pgbouncer/pgbouncer env: - name: DB_HOST value: postgres.production - name: MAX_CLIENT_CONN value: "1000" - name: DEFAULT_POOL_SIZE value: "20" ``` All app pods connect to pgbouncer Service → pgbouncer maintains 20 connections to Postgres. 500 app pods × share 20 real DB connections. Pool modes: • Session: Connection held for entire session. 1:1 unless client disconnects. • Transaction: Connection returned to pool after each transaction. Works for most apps. Doesn't support SET statements, advisory locks. • Statement: Return after each statement. Very aggressive pooling. Incompatible with multi-statement transactions. RDS Proxy (AWS): Managed connection pooler for RDS/Aurora. IAM authentication, no infrastructure to manage. Application-level pool sizing: ```yaml # Spring Boot application.yml spring: datasource: hikari: maximum-pool-size: 10 # keep small when using pgbouncer minimum-idle: 2 connection-timeout: 30000 ```

Essential advanced kubectl commands for production operations. Resource inspection: ```bash # Get all resources in a namespace kubectl get all -n production # Watch pod restarts kubectl get pods -w --field-selector status.phase=Running # Sort pods by restart count kubectl get pods --sort-by=.status.containerStatuses[0].restartCount # Get pods on a specific node kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-1 # Resources using most CPU/memory kubectl top pods --all-namespaces --sort-by=cpu | head -20 kubectl top nodes ``` Quick edits: ```bash # Edit resource directly kubectl edit deployment order-service # Patch without edit kubectl patch deployment order-service -p '{"spec":{"replicas":5}}' # Set image (quick rollout) kubectl set image deployment/order-service api=myregistry/api:1.5.0 # Scale kubectl scale deployment order-service --replicas=10 ``` Debugging: ```bash # Copy files from/to pod kubectl cp production/order-service-abc:/app/logs/error.log ./error.log # Port forward for local access kubectl port-forward deployment/order-service 8080:8080 kubectl port-forward svc/postgres 5432:5432 # Exec into container kubectl exec -it order-service-abc -c main -- /bin/sh # Get events sorted by time kubectl get events --sort-by=.metadata.creationTimestamp -n production # Cluster-wide events kubectl get events -A --sort-by=.lastTimestamp | tail -50 ``` JSONPath and custom output: ```bash # Extract specific fields kubectl get pods -o jsonpath='{.items[*].metadata.name}' kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}\t{.status.capacity.cpu}\n{end}' # Custom columns kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase ``` Bulk operations: ```bash # Delete all evicted pods kubectl get pods -A | grep Evicted | awk '{print $2 " -n " $1}' | xargs kubectl delete pod # Restart all pods in a deployment kubectl rollout restart deployment/order-service ```

Multi-cluster management: Operating multiple Kubernetes clusters — different regions, different environments, different tenants — with consistent policies, configuration, and observability. Why multiple clusters: • Geographic distribution: Clusters in multiple regions for latency and redundancy • Environment isolation: Separate dev, staging, prod clusters (stronger isolation than namespaces) • Risk isolation: Production outage in one region doesn't affect another • Regulatory compliance: Data sovereignty — customer data must stay in specific region • Scale limits: One cluster has practical limits (~5,000 nodes, 150,000 pods) Tools for multi-cluster management: Kubeconfig contexts: ```bash kubectl config get-contexts kubectl config use-context prod-us-east-1 kubectl --context prod-eu-west-1 get pods -n production ``` Kubefed (Federation v2): Propagate resources across clusters. Create a FederatedDeployment → deploys to all member clusters. ArgoCD multi-cluster: ```yaml # Register external clusters argocd cluster add prod-eu-cluster argocd cluster add prod-us-cluster # ApplicationSet deploys to all clusters generators: - clusters: {} # auto-discovers all registered clusters ``` Flux multi-cluster: Flux running in each cluster, each cluster subscribes to its own git path. Karmada: Multi-cluster resource propagation with policy-based scheduling — place workloads across clusters based on affinity, resource availability, and cost. Service mesh federation (Istio): ```yaml # Connect services across clusters # east cluster can call west cluster services via east-west gateway ``` Observability: Thanos (multi-cluster Prometheus federation), Grafana datasources for each cluster, centralized Alertmanager. Kubeconfig management: aws eks update-kubeconfig, gcloud container clusters get-credentials, or tools like kubie, kubeswitch, kubectx for ergonomic context switching.

Kustomize: A Kubernetes-native configuration management tool (built into kubectl since 1.14). Allows customizing Kubernetes YAML without forking or templating — uses strategic merge patches and JSON patches on top of a base configuration. Core concepts: • base: Common resource definitions shared across environments • overlay: Environment-specific customizations applied on top of a base • kustomization.yaml: Declares what to include and how to transform Base kustomization.yaml: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml - ingress.yaml commonLabels: managed-by: kustomize commonAnnotations: app.kubernetes.io/managed-by: kustomize ``` Overlay kustomization.yaml (production): ```yaml bases: [../../base] namespace: production images: - name: myapp newName: myregistry/myapp newTag: "1.5.0" replicas: - name: api count: 10 patchesStrategicMerge: - production-resources.yaml # override CPU/memory limits patchesJson6902: - target: kind: Deployment name: api patch: | - op: replace path: /spec/template/spec/containers/0/env/0/value value: "production" configMapGenerator: - name: app-config literals: - LOG_LEVEL=WARN - ENVIRONMENT=production behavior: merge # merge with base configmap secretGenerator: - name: db-secret envs: [.env.production] # .env file → Kubernetes Secret ``` Apply: ```bash kubectl apply -k overlays/production kubectl diff -k overlays/production kustomize build overlays/production # print rendered YAML ``` Transformers: namePrefix/nameSuffix, namespace override, label transformers, image tag transformers — all applied during rendering without modifying base files.

Ingress controller: Implements the Kubernetes Ingress resource — reads Ingress rules and configures an actual load balancer or reverse proxy. Must be deployed separately (not included in Kubernetes core). How it works: 1. Developer creates Ingress resource (routing rules) 2. Ingress controller watches Ingress resources 3. Controller configures the underlying proxy (Nginx, Envoy, cloud LB) 4. External traffic hits the proxy → routed to Services Popular ingress controllers: nginx-ingress (kubernetes/ingress-nginx): • Most widely used • Configures Nginx reverse proxy • Rich annotation support for rate limiting, auth, rewrites, timeouts • Good for monolithic gateway — one controller for all apps ```yaml annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/rate-limit: "100r/m" nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com/verify" ``` Traefik: • Middleware-based plugin system • Native Kubernetes CRDs (IngressRoute) • Let's Encrypt integration built-in • Good for microservices routing with per-route middleware AWS ALB Controller (AWS Load Balancer Controller): • Creates AWS Application Load Balancers per Ingress • Native AWS integration (WAF, ACM certificates, Cognito auth) • Targets pods directly (no NodePort overhead) Istio IngressGateway: • When you're already using Istio • Uses Gateway + VirtualService resources • Full Istio traffic management for ingress Contour (Envoy-based): • Uses Envoy proxy • HTTPProxy CRD for advanced routing • Good performance, active OSS community Gateway API (future): Replaces Ingress across all controllers. Nginx, Contour, Istio, Traefik all implementing Gateway API support. Choice guide: EKS → AWS ALB Controller. General Kubernetes → Nginx Ingress (simple) or Contour (performance). Service mesh → Istio Gateway. Microservices with complex routing → Traefik.

StatefulSet: Manages stateful applications where each pod instance needs a stable identity, stable storage, and ordered operations. StatefulSet guarantees: 1. Stable network identity: Pods get predictable names: {statefulset}-0, {statefulset}-1, ... 2. Stable persistent storage: Each pod gets its own PVC (not shared). PVC persists across pod deletion and rescheduling. 3. Ordered deployment: Pods created in order (0 → 1 → 2). Pod 1 not started until Pod 0 is Running and Ready. 4. Ordered scaling: Scale down from the highest ordinal (2 → 1 → 0). Ensures primary/leader (usually pod-0) is last to go. 5. Ordered rolling update: Updates from highest to lowest ordinal. ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres spec: serviceName: postgres-headless # required headless service replicas: 3 selector: matchLabels: app: postgres template: metadata: labels: app: postgres spec: containers: - name: postgres image: postgres:16 volumeMounts: - name: data mountPath: /var/lib/postgresql/data volumeClaimTemplates: # PVC per pod - metadata: name: data spec: accessModes: [ReadWriteOnce] storageClassName: fast-ssd resources: requests: storage: 100Gi ``` Pod DNS with headless service: ``` postgres-0.postgres-headless.production.svc.cluster.local postgres-1.postgres-headless.production.svc.cluster.local postgres-2.postgres-headless.production.svc.cluster.local ``` Application-specific roles: • Primary election: Applications like Patroni (Postgres HA) use StatefulSet pod ordinals + etcd/etcd for leader election • Initialization: pod-0 initializes as primary; pod-1, pod-2 join as replicas Update strategy: • RollingUpdate (default): Update pod-2 → pod-1 → pod-0 • OnDelete: Only update pods when manually deleted • partition: Only update pods with ordinal >= partition value (canary updates)

Cluster API (CAPI): A Kubernetes project for declarative cluster lifecycle management — provisioning, upgrading, and operating Kubernetes clusters using the Kubernetes API and CRDs. Concept: Instead of using cloud CLIs or console to create clusters, you declare clusters as Kubernetes resources in a management cluster. Cluster API controllers in the management cluster reconcile to create and manage workload clusters. Key resources: • Cluster: Defines the desired cluster (network CIDRs, control plane endpoint) • Machine: Represents a single node in a cluster • MachineDeployment: Like Kubernetes Deployment, but for nodes • InfrastructureProvider: Cloud-specific (AWSCluster, AzureCluster, vSphereCluster) Provider structure: • Bootstrap provider: Generates cloud-init configuration to install Kubernetes on machines (kubeadm, k3s) • Control plane provider: Manages control plane (KubeadmControlPlane) • Infrastructure provider: Cloud-specific resources (EC2 instances, Azure VMs, vSphere VMs) Example Cluster definition: ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: production-us-east-1 spec: clusterNetwork: pods: cidrBlocks: ["10.0.0.0/16"] controlPlaneRef: kind: KubeadmControlPlane name: production-control-plane infrastructureRef: kind: AWSCluster name: production-aws-cluster --- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster spec: region: us-east-1 sshKeyName: my-ssh-key network: vpc: availabilityZoneUsageLimit: 3 ``` Benefits: • Cluster as code: Cluster definitions in git, version-controlled • Consistent tooling: Same kubectl workflow for cluster operations • Multi-provider: Switch from AWS to GCP by changing the infrastructure provider • Upgrade automation: Machine rolling updates, control plane upgrades via controller Used by: EKS Anywhere, OpenShift, Rancher, and many enterprise Kubernetes platforms.

Finalizers and owner references control resource deletion and garbage collection in Kubernetes. Finalizers: Strings on resource metadata that block deletion. When a resource is marked for deletion (deletionTimestamp set), Kubernetes doesn't delete it until all finalizers are removed. Only the controller responsible for each finalizer should remove it after completing cleanup. ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-data finalizers: - kubernetes.io/pvc-protection # kubelet adds this; prevents deletion while pod uses PVC # After pod stops using PVC, pvc-protection controller removes finalizer → PVC deleted ``` Custom finalizer in an operator: ```go func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { db := &myv1.Database{} r.Get(ctx, req.NamespacedName, db) if db.DeletionTimestamp != nil { // Resource is being deleted if containsString(db.Finalizers, "my.operator/cleanup") { // Perform cleanup: drop database, revoke users r.cleanupDatabase(db) // Remove finalizer — allows deletion to proceed controllerutil.RemoveFinalizer(db, "my.operator/cleanup") r.Update(ctx, db) } return ctrl.Result{}, nil } // Normal reconciliation: add finalizer if !containsString(db.Finalizers, "my.operator/cleanup") { controllerutil.AddFinalizer(db, "my.operator/cleanup") r.Update(ctx, db) } // ... rest of reconciliation } ``` Owner References: Create parent-child relationships between resources. When the owner is deleted, garbage collection automatically deletes the owned resources. ```yaml # Pod owned by ReplicaSet (auto-set by Kubernetes) metadata: ownerReferences: - apiVersion: apps/v1 kind: ReplicaSet name: my-rs uid: abc-123 controller: true blockOwnerDeletion: true ``` Garbage collection: kubectl delete replicaset my-rs → also deletes all pods owned by it. This is how deleting a Deployment cascades to ReplicaSet, then to Pods.

Resource quotas enforce fair resource sharing in a multi-team cluster. Teams can't monopolize cluster resources. ResourceQuota types: 1. Compute quotas: ```yaml kind: ResourceQuota spec: hard: requests.cpu: "40" # total requested CPU limits.cpu: "80" # total CPU limits requests.memory: "80Gi" limits.memory: "160Gi" ``` 2. Object count quotas: ```yaml spec: hard: pods: "100" services: "20" services.loadbalancers: "5" services.nodeports: "0" # block NodePort (security) persistentvolumeclaims: "30" secrets: "100" configmaps: "100" deployments.apps: "50" ``` 3. Storage class quotas: ```yaml spec: hard: fast-ssd.storageclass.storage.k8s.io/requests.storage: "2Ti" standard.storageclass.storage.k8s.io/requests.storage: "5Ti" ``` 4. Priority class quotas (scope): ```yaml spec: scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: [high-priority] hard: pods: "10" # only 10 high-priority pods per namespace ``` LimitRange (defaults and max/min): ```yaml kind: LimitRange spec: limits: - type: Container defaultRequest: cpu: 200m memory: 256Mi default: cpu: 500m memory: 512Mi max: cpu: "8" memory: 16Gi ``` Practical multi-team setup: 1. Create namespace per team: team-orders, team-payments, team-catalog 2. Apply ResourceQuota: proportional to team allocation 3. Apply LimitRange: sensible defaults prevent zero-resource pods 4. RBAC: teams can only manage their namespace 5. Monitoring: Grafana dashboard showing quota usage per team — alert before exhaustion Checking quota: ```bash kubectl describe resourcequota -n team-orders # Shows: hard limits and current usage ```

Topology-aware routing: Routes service traffic preferentially to endpoints in the same topology zone (e.g., same availability zone), reducing cross-zone traffic latency and costs. Why topology matters: • Cross-AZ traffic: Data transfer between AZs costs money (e.g., ~$0.01/GB on AWS) • Latency: Same-AZ traffic is faster than cross-AZ • Network bandwidth: Keeping traffic local reduces inter-AZ bandwidth consumption Topology-aware routing (K8s 1.27 stable): Enabled by annotating the Service: ```yaml apiVersion: v1 kind: Service metadata: name: order-service annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: order-service ports: - port: 8080 ``` With Auto mode: • EndpointSlice controller adds topology hints (which zone each endpoint is in) • kube-proxy uses hints to prefer endpoints in the same zone as the requesting pod • Falls back to all endpoints if local zone doesn't have enough healthy endpoints (threshold: 33%) Requirements for topology-aware routing: • Nodes must have topology.kubernetes.io/zone labels • Sufficient replicas across zones (otherwise Kubernetes can't ensure local routing) • Service must not be headless (ClusterIP service) HPA with topology awareness: Deploy replicas proportional to zone size: ```yaml # Topology spread constraints ensure even zone distribution topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: order-service ``` Traffic split in a 3-AZ setup: • Without topology routing: 100 pods across 3 AZs → 2/3 traffic crosses zones • With topology routing: Each zone's pods only receive local zone traffic → 0 cross-zone traffic Istio locality load balancing: Istio's service mesh implementation of topology-aware routing with configurable failover policies between zones and regions.

Argo Workflows: A Kubernetes-native workflow engine for defining and running complex multi-step workflows as Directed Acyclic Graphs (DAGs). Each step runs as a Kubernetes pod. When Kubernetes CronJob is insufficient: • Multi-step pipelines with dependencies: Step B runs after Step A succeeds • Parallel fan-out/fan-in: Process 1000 items in parallel, aggregate results • Conditional branching: If ML model accuracy > threshold, deploy; otherwise alert • Complex retry logic per step • Human approval gates (Argo Events integration) • Long-running workflows with state Argo Workflow example (data pipeline): ```yaml apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: data-pipeline- spec: entrypoint: pipeline templates: - name: pipeline dag: tasks: - name: extract template: extract-data - name: transform dependencies: [extract] # waits for extract template: transform-data arguments: parameters: - name: input-file value: "{{tasks.extract.outputs.parameters.output-file}}" - name: validate dependencies: [transform] template: validate-data - name: load dependencies: [validate] template: load-to-warehouse - name: extract-data container: image: data-extractor:latest command: [python, extract.py] outputs: parameters: - name: output-file valueFrom: path: /tmp/output-path.txt ``` Argo vs CronJob: • CronJob: Run one pod on a schedule. No multi-step, no DAG, no parallel fan-out. • Argo Workflow: Multi-step DAG, parallel execution, artifacts passing between steps, retry per step, visual UI, parameter passing. Argo Events: Triggers Argo Workflows from external events (webhook, Kafka message, S3 upload, GitHub push) — not just cron. Argo CD: Separate tool — GitOps. Not related to Argo Workflows despite the name prefix.

Kubernetes clusters can be expensive. Systematic cost optimization requires visibility, right-sizing, and architectural changes. 1. Cost visibility: • OpenCost (formerly Kubecost): Breaks down Kubernetes cost by namespace, pod, label. Shows actual AWS/GCP costs attributed to each workload. ```bash kubectl cost namespace production # cost per namespace kubectl cost pod -n production # cost per pod ``` • Cloud provider cost allocation tags: Tag nodes by team, environment → see cluster cost in AWS Cost Explorer. 2. Right-sizing (biggest impact): • VPA recommendations: Run VPA in Off mode for 1 week, view recommendations, right-size requests • Over-provisioned pods: requests >> actual usage → waste money on reserved but unused capacity • Use Goldilocks: Visualizes VPA recommendations across all workloads 3. Spot/Preemptible nodes: ```yaml # Tolerate spot nodes (70% cheaper than on-demand) tolerations: - key: "node.kubernetes.io/spot" operator: Exists effect: NoSchedule ``` • Use spot for stateless, restartable workloads (batch jobs, worker pods) • On-demand for stateful apps and critical services • Karpenter: Automatically selects cheapest instance type across spot and on-demand 4. Scale to zero with KEDA: Scale idle workloads to 0 replicas — only run when there's work. 5. Node bin-packing: Pack pods tightly to reduce node count. Use MostAllocated scheduler scoring (opposite of default LeastAllocated). 6. Cluster bin-packing: Cluster Autoscaler scale-down consolidates pods onto fewer nodes. Karpenter consolidation: Actively moves pods to pack them on fewer nodes. 7. Namespace-level quotas: Prevent teams from over-provisioning by enforcing ResourceQuotas. 8. Storage optimization: • Delete unused PVCs (orphaned) • Use cheaper storage classes for non-critical data • Right-size PVC storage requests 9. Egress cost reduction: Topology-aware routing reduces cross-AZ data transfer charges.

High availability in Kubernetes means designing pod placement to survive node failures, zone failures, and region failures. Failure domain levels: • Node: Single node failure (hardware, OS, kubelet) • Zone: Entire availability zone failure (power outage, network partition) • Region: Entire cloud region outage (rare but happens) HA design principles: 1. Multiple replicas (obvious but necessary): ```yaml spec: replicas: 6 # survive losing any single zone in 3-AZ cluster ``` 2. Zone spread with pod anti-affinity: ```yaml spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: {app: api} topologyKey: topology.kubernetes.io/zone # never 2 pods in same zone ``` 3. Topology spread constraints (more flexible): ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: {app: api} - maxSkew: 2 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: {app: api} ``` 4. PodDisruptionBudget (survive planned disruptions): ```yaml kind: PodDisruptionBudget spec: minAvailable: 2 # at least 2 pods always running ``` 5. Health checks (route away from failed pods): ```yaml readinessProbe: httpGet: path: /health port: 8080 failureThreshold: 3 # remove from load balancing after 3 failures ``` 6. Graceful shutdown (survive rolling updates): ```yaml lifecycle: preStop: exec: command: [sleep, "15"] # drain in-flight requests terminationGracePeriodSeconds: 60 ``` 7. Resource guarantees (prevent eviction): ```yaml resources: requests: cpu: 200m memory: 256Mi limits: cpu: 200m # Guaranteed QoS — last to be evicted memory: 256Mi ``` Multi-region HA: Multiple clusters + global load balancer (AWS Route53, Cloudflare) with health checks. Active-active or active-passive depending on data consistency requirements.

Karpenter: An open-source, flexible, high-performance Kubernetes node provisioner (developed by AWS, now CNCF). Directly provisions cloud instances for pending pods — faster and more cost-efficient than the Cluster Autoscaler. Cluster Autoscaler limitations: • Node group-based: CA scales pre-defined Auto Scaling Groups. Instance type fixed at ASG creation. • Slow: CA calls ASG to scale, ASG provisions EC2 — 2-5 minutes. • Fixed node types: Can't pick the cheapest available spot instance on demand. • No consolidation: Doesn't move pods to pack them onto fewer nodes. Karpenter advantages: • Watches pending pods directly via the Kubernetes API • Selects the optimal instance type for each batch of pending pods (right-size, not just "same type as the ASG") • Directly calls EC2 API to provision instances (60-90 seconds) • Picks cheapest Spot instance from allowed pool automatically • Consolidation: Actively moves pods off underutilized nodes and terminates them Karpenter NodePool: ```yaml apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: general-purpose spec: template: spec: nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: default requirements: - key: kubernetes.io/arch operator: In values: [amd64, arm64] - key: karpenter.sh/capacity-type operator: In values: [spot, on-demand] - key: node.kubernetes.io/instance-type operator: In values: - m5.xlarge - m5.2xlarge - m6i.xlarge - m6g.xlarge # Graviton (arm64, cheaper) limits: cpu: "1000" memory: "4000Gi" disruption: consolidationPolicy: WhenUnderutilized consolidateAfter: 1m ``` EC2NodeClass: ```yaml apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: default spec: amiFamily: AL2 role: "KarpenterNodeRole" subnetSelectorTerms: - tags: {karpenter.sh/discovery: my-cluster} securityGroupSelectorTerms: - tags: {karpenter.sh/discovery: my-cluster} ```

RBAC issues: "Error from server (Forbidden): pods is forbidden: User cannot get resource pods" — diagnosing and fixing permission denials. Step 1 — Check current permissions: ```bash # What can the current user do? kubectl auth can-i get pods -n production kubectl auth can-i create deployments -n production kubectl auth can-i "*" "*" --all-namespaces # cluster admin check # Check permissions for a specific user/serviceaccount kubectl auth can-i get pods -n production \ --as=jane@company.com kubectl auth can-i get pods -n production \ --as=system:serviceaccount:production:order-service-sa ``` Step 2 — Find all roles bound to a user: ```bash # Show all ClusterRoleBindings for a user kubectl get clusterrolebindings -o json | jq '[ .items[] | select(.subjects[]? | select(.name == "jane")) | {name: .metadata.name, role: .roleRef.name} ]' # Show all RoleBindings in a namespace kubectl get rolebindings -n production -o yaml ``` Step 3 — Inspect role permissions: ```bash kubectl describe clusterrole view kubectl describe role my-custom-role -n production ``` Step 4 — Enable audit logging to trace requests: ```yaml # Audit policy - log all forbidden requests rules: - level: RequestResponse verbs: ["*"] resources: ["*"] users: ["jane@company.com"] ``` Step 5 — Fix: Create appropriate Role and RoleBinding: ```yaml kind: Role metadata: name: pod-reader namespace: production rules: - apiGroups: [""] resources: [pods, pods/log] verbs: [get, list, watch] --- kind: RoleBinding metadata: name: jane-pod-reader namespace: production subjects: - kind: User name: jane@company.com apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` Common RBAC mistakes: • Wrong API group: pods are in "" (core), deployments in "apps" — must specify correct apiGroups • Namespace scope: Role only works in its namespace. Use ClusterRole for cluster-wide resources. • ServiceAccount namespace: must specify correct namespace in subject: system:serviceaccount:NAMESPACE:NAME

Argo Rollouts: A Kubernetes controller that provides advanced deployment strategies (canary, blue-green) with automated analysis and rollback based on metrics. Why Argo Rollouts over native Deployments: • Native K8s rolling update: all-or-nothing rollout, no traffic splitting, no metric-based rollback • Argo Rollouts: gradual traffic shifting, metric analysis, automatic rollback, manual pause points Rollout with canary analysis: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: order-service spec: replicas: 10 revisionHistoryLimit: 3 selector: matchLabels: app: order-service strategy: canary: canaryService: order-service-canary # separate canary Service stableService: order-service-stable # separate stable Service trafficRouting: nginx: stableIngress: order-service-ingress steps: - setWeight: 5 # 5% to canary - pause: {duration: 10m} - analysis: # run analysis during rollout templates: - templateName: success-rate - setWeight: 20 - pause: {duration: 5m} - setWeight: 50 - pause: {} # manual approval required - setWeight: 100 ``` AnalysisTemplate: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 1m successCondition: result[0] >= 0.95 # fail if < 95% success failureLimit: 3 provider: prometheus: address: http://prometheus:9090 query: | sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m])) / sum(rate(http_requests_total{service="{{args.service-name}}"}[2m])) ``` Automatic rollback: If analysis fails (success rate < 95%), Rollout automatically aborts and routes all traffic back to stable. Manual control: ```bash kubectl argo rollouts promote order-service # advance past pause kubectl argo rollouts abort order-service # rollback immediately kubectl argo rollouts status order-service # live status ```

Service mesh: Infrastructure layer for service-to-service communication. Intercepts all network traffic between services via sidecar proxies, providing observability, traffic management, and security without application code changes. Core service mesh capabilities: • mTLS: Mutual authentication between all services • Distributed tracing: Automatic trace context propagation • Traffic management: Canary, A/B testing, retries, timeouts, circuit breaking • Observability: Automatic per-service metrics (request rate, latency, error rate) • Access control: Service-to-service authorization policies Popular service meshes: • Istio: Full-featured, uses Envoy sidecar, most adopted • Linkerd: Lightweight, simpler, better performance, Rust-based proxy • Consul Connect: HashiCorp, integrates with Consul service discovery • AWS App Mesh: Managed, uses Envoy, ECS/EKS integration • Cilium Service Mesh: eBPF-based, sidecarless option Service mesh costs: • Resource overhead: Envoy sidecar per pod (50-100MB memory, 5-15ms latency) • Operational complexity: New control plane to manage, update, and secure • Learning curve: VirtualService, DestinationRule, PeerAuthentication, AuthorizationPolicy • Debugging harder: Two layers of networking (app + mesh) Do you NEED a service mesh? NO, if: • Small cluster (< 20 services) • Team unfamiliar with service mesh concepts • Low security requirements (no mTLS needed internally) • Simple traffic management (no canary, no A/B testing needed) • Observability already handled at application level YES, if: • Many services requiring uniform mTLS • Compliance requires encrypted inter-service traffic • Complex traffic routing (canary across many services) • Cross-language services where adding retry/circuit-break logic per service is impractical • Need service graph visualization (Kiali) Alternatives to full mesh: Cilium network policies for security, application-level resiliency (Resilience4j), Argo Rollouts for traffic management.

Kubernetes production best practices — the definitive checklist. Cluster setup: • Multi-AZ control plane (3+ control plane nodes across AZs) • Worker nodes across 3 AZs for workload HA • Private API server endpoint + network policy to restrict access • etcd backups every 6 hours, stored in S3 • Node OS: container-optimized (Bottlerocket, COS, Flatcar) • Regular cluster upgrades (stay within 1-2 minor versions of latest) Workload configuration: • Always set resource requests AND limits (Guaranteed QoS) • Define liveness, readiness, and startup probes • Use exec form CMD/ENTRYPOINT (not shell form) for proper signal handling • terminationGracePeriodSeconds ≥ application shutdown time • preStop hook with sleep to handle load balancer deregistration race High availability: • minReplicas ≥ 2 for any service that must be available • PodDisruptionBudget on every production service • podAntiAffinity or TopologySpreadConstraints for zone distribution • HPA for dynamic scaling (+ KEDA for event-driven) Security: • Pod Security Standards: restricted for production namespaces • Run as non-root (runAsNonRoot: true, runAsUser: 1000+) • Read-only root filesystem + specific tmpfs mounts • Drop ALL capabilities, add only what's needed • Network policies: default deny all, explicit allow • External Secrets Operator or Vault for secret management • IRSA/Workload Identity (never long-lived credentials in pods) • Regular image scanning (Trivy in CI + Continuous scanning) • Audit logging enabled and shipped to central SIEM Observability: • Prometheus + Grafana for metrics • Loki or ELK for logs • Jaeger/Tempo for distributed tracing • Alertmanager rules for critical paths • SLOs defined and tracked per service CI/CD and deployments: • GitOps (ArgoCD/Flux) — no direct kubectl in production • Image tags pinned to SHA — never :latest in prod • Rolling update with maxUnavailable: 0 • Rollback tested regularly Cost: • VPA recommendations reviewed regularly • Spot instances for batch/stateless workloads • Cluster Autoscaler or Karpenter for node rightsizing • ResourceQuota per team namespace