How does Netflix run canary releases of its microservices?

An automated Spinnaker pipeline rolls the new version to a few instances that get 1–5% of traffic, compares their metrics against the stable version, and then automatically widens the rollout or stops and rolls back.

* Jenkins builds, Spinnaker deploys a small canary, metrics compare it against the stable version, and the pipeline promotes or rolls back on its own. *

1. Pipeline: Spinnaker. Netflix built Spinnaker, an open-source tool for multi-cloud deployments that supports canary releases, blue/green (red/black) deployments and rolling updates.
2. Canary phase. The new version goes to a small number of instances, which receive only a fraction of the traffic (around 1–5%). Their performance is compared with the current stable version.
3. Monitoring and metrics. Netflix's own tool Atlas, along with Prometheus and Grafana, tracks error rate, latency and system load.
4. Automated decision. If the canary runs stably, it is rolled out to more instances automatically. If not, the rollout is stopped or rolled back. No person has to watch dashboards and decide.

The surrounding stack includes Jenkins (CI builds), Kubernetes (container orchestration) and Spinnaker (deployment).

On top of this, Netflix uses Chaos Monkey to test whether the new version holds up when instances fail (see the next card).

Go deeper:

• Spinnaker — Netflix's open-source multi-cloud continuous delivery platform.
• Spinnaker — Automated Canary Analysis — the canary-judging stage Netflix built into it.