Version Skew: Running Mixed Versions Without Breaking
Version skew is the normal state during rollouts: old and new code run side by side. Designing for it prevents silent incompatibilities and failed rollbacks.
Definition
Version skew is the condition where components of a system run different versions at the same time, whether during a rolling deployment, across regions, or between a control plane and its clients.
Context and scope
Any system that deploys incrementally experiences skew; APIs, schemas, and protocols must remain compatible across the supported window or upgrades become outages.
Common misconceptions
- Skew lasts seconds — skew lasts for the whole rollout and for every client that upgrades slowly
- Skew only affects APIs — message formats, caches, and feature flags also carry version assumptions
- Rollback restores the old world — data written by the new version may be unreadable to the old one
Anti-patterns
- Changing a shared schema and all its readers in the same release
- Assuming all clients upgrade together
- Testing only the new version against the new version
Related concepts
Authoritative source
https://kubernetes.io/docs/setup/release/version-skew-policy/
Mixed versions are normal
During a rolling deployment, some instances run the old build and some run the new one. Across regions, a failover can pair a new control plane with old workers. Mobile clients upgrade over months. Version skew is not an anomaly to eliminate; it is the steady state of any distributed system, and it must be designed for explicitly.
Compatibility windows
Every interface needs a stated compatibility window: how many versions can coexist and for how long. Kubernetes, for example, publishes a skew policy that defines which control plane and node versions may differ. Internal APIs need the same thing, written down, because “we always deploy together” stops being true the moment there are two teams.
Data is the hard part
Code can often run both versions; data cannot. A new version that writes a new column, a new enum value, or a new message field creates state the old version may not understand. Use expand-and-contract migrations: add first, deploy readers, backfill, then remove in a later release. For example, writing a status value that the old code rejects will turn a rollback into an outage.
Testing skew
Test the combinations you actually support: new against old, old against new, and both against shared state. Contract tests catch wire-level breaks; replaying recorded traffic catches behavioural ones. Add a skew case to the release checklist, because the default test suite only exercises matched versions.
Compared to atomic upgrades
Atomic upgrades avoid skew by switching everything at once, at the cost of downtime or a complex cutover. Incremental upgrades accept skew to keep availability. The trade is only safe when compatibility is verified rather than assumed, and when rollback is tested against data written by the newer version.