Data Schema Evolution Governance Workflow

Evolve event and table schemas through compatibility classes, consumer-aware reviews, and dual-write transition windows instead of breaking swaps.

intermediate~30 mindata-pipelineschemagovernance

Objective

Change data schemas without breaking downstream consumers, using declared compatibility levels and staged expansion-contraction transitions.

Environment

Schemas under version control with a registry or migration tool
A list of active consumers per schema
A review process with both producer and consumer owners

Steps

1. Declare the compatibility class

Classify the change first: backward-compatible (add optional field), forward-compatible (ignore unknown), or breaking, which dictates the process weight.

Expected state: The change carries a compatibility label before any review happens.

2. Review with the consumer map

For anything beyond additive, review against the consumer inventory, confirming each consumer's tolerance for the change explicitly.

Expected state: Affected consumers are named and have acked, not assumed fine.

3. Expand before you contract

Implement breaking changes as expansion (new field or table alongside old) with a transition window where both exist and producers dual-write.

Expected state: Old consumers keep working during the window; new field is authoritative and backfilled.

4. Migrate consumers on their schedule

Move each consumer to the new schema inside the window, verifying with dual-read comparisons that outputs match expectations.

Expected state: Consumers migrate one by one with evidence, and the old path stays until the last one moves.

5. Contract after the window closes

Only when all consumers migrated and the window elapsed, remove the old field or table in a final, announced contraction change.

Expected state: The removal change references the completed migration record.

Expected outcome

Schema changes that never break live consumers, with breaking changes executed as staged expansions and contractions instead of flag-day swaps.

Verification

No consumer incident correlates with a schema change; breaking changes show expansion, migration evidence per consumer, and a dated contraction in git history.

Schemas are contracts with many signatories

A schema change feels local to the producer but executes across every consumer. Declaring compatibility class, checking the consumer map, and staging breaking changes as expand-migrate-contract turns a flag day into a non-event.

event: order.created  v3 (breaking)
compatibility: BREAKING (renames amount_cents)
plan:
  1. expand  v4: add total_cents (dual-write)
     consumers on v3 keep working
  2. migrate consumers (their schedule):
     analytics   10-24  dual-read diff = 0
     billing     10-29  dual-read diff = 0
     fraud       11-02  dual-read diff = 0
  3. contract v5 (11-09): drop amount_cents
     references migration record #447

The consumer map is the review’s input

Compatibility is not a property of the change alone but of the change against actual consumers: adding a required field is additive in syntax but breaking for a strict validator downstream. Reviewing with the inventory, and getting explicit acks, replaces hoping with knowing.

Dual-write windows make migration observable

During expansion both old and new representations exist, so each consumer can dual-read and compare outputs before switching. That comparison is the evidence the migration was correct, and it is only available if the window exists.