Configuration Drift: When Reality Stops Matching the Repository
Configuration drift is the gap between declared and actual state. Detecting it early keeps infrastructure reproducible and incidents explainable.
Definition
Configuration drift is any difference between the state defined in version control or an infrastructure-as-code template and the state actually running. It accumulates through manual hotfixes, console edits, and partial rollbacks.
Context and scope
Drift is inevitable in long-lived systems; the question is whether you detect it within minutes or discover it during the next incident, when the runbook no longer matches reality.
Common misconceptions
- Drift only affects servers — it affects policies, identity roles, DNS records, and feature configuration
- Immutable infrastructure eliminates drift — it limits blast radius, but images and settings still drift
- Detection is enough — without reconciliation, dashboards fill with permanent warnings
Anti-patterns
- Fixing production by hand and never writing the change back to code
- Alerting on drift without an owner or remediation path
- Treating drift alerts as noise and muting them permanently
Related concepts
Authoritative source
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/detect-drift-stack.html
Declared versus actual
Every managed system has two states: what the repository says should exist, and what is actually running. In a young system they match. Over time, someone edits a security group during an incident, a console change bypasses review, and a partial rollback leaves half a cluster on the old template. That gap is configuration drift.
How drift accumulates
Drift rarely arrives in one dramatic act. It comes from hotfixes applied under pressure, from manual scaling that the template never learns about, and from resources created outside the pipeline. Each change is individually reasonable and collectively corrosive: the template stops describing production, so nobody trusts it, and the next change is made by hand as well.
Why it matters at the worst moment
The cost shows up during recovery. A runbook assumes a setting that no longer exists, a replacement environment is built from a template missing three emergency patches, and an audit cannot explain who changed a policy. Drift turns a documented system into folklore, and folklore fails under stress.
Detection and reconciliation
Detection compares declared and actual state on a schedule and after every deploy. Detection alone is not enough: every finding needs an owner, a severity, and a path to either reconcile the resource or update the code. For example, a nightly drift report that opens a ticket for each difference, with the option to accept the change back into the repository, keeps the two states converging instead of diverging.
Compared to immutability
Immutable infrastructure replaces instances instead of mutating them, which removes a large class of drift but not all of it: images, policies, DNS, and runtime configuration can still diverge. Treat immutability as a strong reducer, not a proof. The remaining surface still needs detection, because an unexamined assumption is exactly what breaks during an incident.