Environment Drift Detection and Remediation Workflow
Detect configuration drift between declared and actual state on a schedule, classify each drift as legitimate or rot, and remediate accordingly.
Objective
Keep environments matching their declared state by detecting drift continuously, absorbing legitimate change into declarations, and killing undocumented rot.
Environment
- Infrastructure as code covering the environments
- A drift-detection tool or state-diff capability
- Authority to apply or update declarations
Steps
1. Detect drift on a schedule
Run the state diff between declared and live configurations daily, capturing every divergence with resource and attribute detail.
Expected state: A drift report exists listing each divergence with its resource.
2. Classify each drift
For every divergence, determine whether it is legitimate (an emergency change that must be back-ported into code) or rot (untracked change to be reverted).
Expected state: Each drift entry carries its classification and the decision's owner.
3. Absorb legitimate change into declarations
For legitimate drift, update the infrastructure code to match reality, preserving the change's history through review.
Expected state: Emergency changes exist in code within days, not as orphaned live state.
4. Revert rot mechanically
For undocumented drift, apply the declared state, restoring the environment to what code says, with the revert recorded.
Expected state: Undocumented changes disappear within the remediation window.
5. Close the loop on drift rate
Track drift volume by source (emergency hotfixes, manual debugging, tooling), attacking the largest source structurally.
Expected state: Drift volume trends down as its root sources are addressed.
Expected outcome
Environments whose live state matches declared state within a remediation window, with legitimate change absorbed into code and drift sources shrinking.
Verification
The daily drift report converges near zero; emergency hotfixes appear in code within the policy window; drift sources show structural decline.
Drift is a decision recorded in the wrong place
Every configuration change made live is a decision; the only question is whether it lands in code or in memory. Scheduled detection with a classify-absorb-revert loop moves all decisions into code within days.
daily drift report (2026-10-22):
prod-cluster:
sg-web: ingress rule 0.0.0.0/0:9200
(declared: absent) classification?
-> ROT (debug session, Oct 03) -> revert
api-deploy: replicas 14
(declared: 12) classification?
-> LEGITIMATE (incident INC-2966
capacity bump) -> absorb into tf
remediation:
rot: terraform apply (sg reverted)
legit: PR #5404 updates replicas, links INC-2966
drift by source (30d):
manual debugging 61% -> prod access review tightened
emergency hotfix 29% -> hotfix SOP gains "absorb" step
tooling 10%The open debugging port is how breaches happen
The classic drift story: an engineer opens a security-group port to debug, closes the session, forgets the port. Detection with revert closes it mechanically; tightening production access attacks the source. Both are needed, and only one of them is automatic.
Absorbing legitimate drift honors reality
Blindly reverting everything breaks the environment that someone correctly fixed at 2 a.m. Classification respects that: the incident-driven replica bump gets absorbed into code with its paper trail, so the next apply does not undo a deliberate decision.