Schema Evolution
Schema evolution changes data or event schemas over time while old and new producers and consumers continue to interoperate.
Definition
Schema evolution is the controlled process of changing a schema without breaking existing readers or writers. Changes are classified by compatibility: adding an optional field is usually backward compatible, while removing a required field or changing its type is not. Versioned schemas, compatibility checks, and upcasting translate old data into the current representation.
Context and scope
Schema evolution applies to event streams, database tables, API payloads, and analytics data. It is especially important in event-driven systems where consumers deploy independently and historical events are replayed.
Common misconceptions
- Adding a field is always safe — a new required field can break consumers that construct the event
- A schema registry removes the need for coordination — it enforces structural rules, not business semantics
- Old events can be rewritten — history is immutable; use upcasting or a new event version instead
Anti-patterns
- Renaming a field in place without a compatibility period
- Changing a field's meaning while keeping its name
- Deploying a producer before consumers can handle the new shape
Related concepts
Related examples
Authoritative source
https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html
Classify the change
Before changing a schema, classify it: additive and optional, additive and required, type change, rename, or removal. Additive optional fields are usually backward compatible; required additions, renames, and removals need a migration period. The classification determines the rollout order and whether old data must be upcast.
{
"version": 2,
"fields": [
{ "name": "id", "type": "string" },
{ "name": "total", "type": "long" },
{ "name": "currency", "type": "string", "default": "USD" }
]
}The example adds an optional field with a default. A consumer that does not know the field ignores it; a producer that omits it still satisfies the consumer.
Roll out in a compatible order
For a backward-compatible addition, deploy consumers first or ensure they tolerate unknown fields, then producers. For a breaking change, run both versions during a transition: write the old and new shapes, migrate readers, then remove the old shape. This is the same expand-contract discipline used for database columns.
Preserve history with upcasting
Event stores keep old events. Instead of rewriting them, an upcaster converts an old event version into the current in-memory representation when reading. Upcasters must be deterministic and tested against real historical payloads. Keep the original event unchanged for audit and replay.
When to use it
Use explicit schema evolution whenever independent teams or long-lived data share a schema. The advantage is safe, incremental change; the drawback is versioning and compatibility work. Compared to an unversioned JSON contract, schema evolution is slower to change but far less likely to break a consumer silently.