Alert on Runtime Config Drift from Git

Diff live ConfigMaps against the manifests in Git, alert on differences, and stop emergency edits from becoming permanent.

~25 minkubernetesconfigurationmonitoring

Task

Know within minutes when someone edits a live ConfigMap instead of merging a change to the repository.

When to use this

During an incident an engineer raises a pool size with kubectl edit. The fix works, the ticket closes, and the next deploy silently reverts it, causing a second outage.

Prerequisites

kubectl 1.27 or newer with server-side apply support
Read access to the namespace from CI
A rendered manifest of the expected state

Steps

1. Diff the rendered manifests against the live cluster.

kubectl diff -f rendered/ -n prod

Expected output: No output and exit code 0 when the cluster matches Git; exit code 1 with a diff when it does not.

2. Turn the diff into an alert instead of failing the job.

#!/usr/bin/env bash
set -uo pipefail
kubectl diff -f rendered/ -n prod > /tmp/drift.diff 2>/tmp/drift.err
code=$?
if [ "$code" -eq 1 ]; then
  curl -fsS -X POST "$ALERT_WEBHOOK" --data-binary @/tmp/drift.diff
elif [ "$code" -gt 1 ]; then
  echo "diff failed: $(cat /tmp/drift.err)" >&2
  exit 1
fi

Expected output: A drifted object posts the exact field diff, while an unreachable cluster is reported as an error.

3. Filter fields that controllers legitimately rewrite before alerting.

kubectl diff -f rendered/ -n prod | grep -vE '^[+-][[:space:]]+(resourceVersion|uid|generation|managedFields)'

Expected output: Server-assigned metadata noise disappears and real spec changes remain.

4. Give emergency edits a path back into Git so the alert has an owner.

Expected output: Every runbook emergency edit links to a pull request template, and the change lands in Git within one day.

Troubleshooting

Problem

The diff always shows changes

Solution

A controller or mutating webhook rewrites the object. Confirm the mutation is expected, then exclude that field from the comparison.

Problem

kubectl diff needs broader permissions than expected

Solution

Scope the ServiceAccount to the namespace and the resource types you own, and grant only get and list on those.

Limitations

kubectl diff does not cover resources that were created outside Git.
Polling every few minutes can miss an edit that is reverted before the check runs.

The cluster is a second source of truth

Git is supposed to be the source of truth, but kubectl, a Helm rollback, or a console session can change the live object without touching it. The cluster then holds a configuration that no repository explains, and the next automated sync removes it. That is how an emergency fix becomes a repeat incident. A drift check compares the rendered expected state with the live object and reports differences while the person who made the change still remembers why.

Diff, then route the result

kubectl diff performs a server-side comparison and exits 0 when there is no difference, 1 when there is, and greater than 1 for an error. Those three outcomes map directly onto a monitoring job: silence, alert, and page the platform team. Do not treat an error as drift, because a broken credential or an unreachable API server would then look like a configuration change. Send the diff itself in the alert so the responder does not need cluster access to understand it.

kubectl diff -f rendered/ -n prod > /tmp/drift.diff 2>/tmp/drift.err
code=$?
[ "$code" -eq 1 ] && curl -fsS -X POST "$ALERT_WEBHOOK" --data-binary @/tmp/drift.diff

Reduce noise, keep the signal

Some fields change constantly: resource versions, managed fields, and status subresources. Filter them out of the comparison, but do so with an explicit list rather than excluding whole object types, because a filter that hides spec changes defeats the check. Run the diff often enough to catch edits before the next deploy, and make the response path routine: either revert the live change or merge it into Git the same day. A drift alert that nobody actions becomes background noise and stops protecting anything.