Canary Deployment
IntermediateRoute a small percentage of traffic to the new version, observe error rate and latency, then gradually increase to 100% or roll back based on signals.
Overview
Canary deployment reduces release risk by exposing the new version to a small, representative slice of real traffic before a full rollout. The name comes from the "canary in a coal mine" — if the canary dies (error rate spikes), retreat before harming everyone. In Kubernetes, a canary is typically implemented with two Deployments sharing one Service via weighted routing (Nginx Ingress, Argo Rollouts, or Istio VirtualService). Automation is key: a progressive delivery controller (Argo Rollouts, Flagger) watches Prometheus metrics and automatically promotes the canary to 100% or rolls it back when it breaches thresholds for p99 latency, error rate, or business KPIs.
Argo Rollouts canary strategy
Argo Rollouts is the standard Kubernetes progressive delivery controller. Define steps for traffic weight and pauses; Rollouts watches analysis metrics and promotes or aborts automatically.
# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: order-service
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5 # 5% canary traffic
- pause: {duration: 2m}
- setWeight: 20
- pause: {duration: 5m}
- setWeight: 50
- pause: {duration: 5m}
- setWeight: 100 # full promotion
analysis:
templates:
- templateName: http-error-rate
startingStep: 1
canaryService: order-service-canary
stableService: order-service-stable
selector:
matchLabels:
app: order-serviceAnalysisTemplate for automated promotion/rollback
Argo Rollouts evaluates AnalysisTemplates using Prometheus queries. If the success rate drops below threshold the rollout is automatically aborted and traffic reverts to the stable version.
# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: http-error-rate
spec:
metrics:
- name: success-rate
interval: 60s
successCondition: result[0] >= 0.99 # 99% success rate
failureLimit: 3
provider:
prometheus:
address: http://prometheus:9090
query: |
sum(rate(http_requests_total{
app="order-service", status!~"5.."
}[2m])) /
sum(rate(http_requests_total{
app="order-service"
}[2m]))Simple Nginx Ingress canary (no Argo)
For teams without Argo, Nginx Ingress supports canary routing via annotations. A second Ingress on the canary Service receives the configured percentage of traffic.
# canary ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: order-service-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10" # 10% traffic
spec:
rules:
- host: api.example.com
http:
paths:
- path: /orders
pathType: Prefix
backend:
service:
name: order-service-canary
port:
number: 8080
# rollback: set canary-weight to "0"
# promote: set canary-weight to "100" then remove canary ingressKey Points to Remember
- 1Start with a small canary slice (1–5%) and use real production traffic — synthetic testing misses real user behavior.
- 2Define success criteria before deploying: p99 latency, error rate, and business metrics (conversion, revenue).
- 3Always have an automated rollback trigger — manual monitoring of canaries at 2 AM is not a strategy.
- 4Canary differs from blue-green: canary is gradual traffic shifting; blue-green is an instant full cutover.
- 5Stateful services (databases) require extra care — canary may run a newer schema against shared state.
- 6Header-based canary routing (X-Canary: true) is useful for internal testing before enabling percentage-based routing.
Interview Questions
Sign in to ask AriaWhat is the difference between a canary deployment and a blue-green deployment?
How would you implement automatic rollback if the canary's error rate exceeds 1%?
How do you handle database schema changes during a canary rollout where old and new code run simultaneously?
What metrics would you monitor during a canary rollout and why?
Explain how Argo Rollouts differs from a standard Kubernetes Deployment for canary releases.
Ask Aria about Canary Deployment
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.