Data Freshness
Data freshness measures how current a dataset is: the lag between when the world changed and when the consumer sees it.
Definition
Data freshness is the age of the newest record in a dataset relative to the event it represents. It is expressed as an SLA — for example, "orders are no later than 15 minutes behind the source" — and monitored as an actual measurement, not a promise. Freshness differs from latency: latency is the time to produce one update, while freshness is how stale the whole table is when a consumer reads it.
Context and scope
Central to any pipeline that feeds dashboards, fraud checks, or models, where decisions depend on how recently the data was updated.
Common misconceptions
- Freshness equals latency — a fast pipeline that runs rarely still serves stale data; freshness is about the newest available record
- Freshness is a one-time property — it varies by table, by hour, and under load, so it must be measured continuously
- Everything must be real-time — freshness should match the decision: a monthly report does not need second-level lag
Anti-patterns
- Reporting a freshness SLA nobody measures, so staleness is discovered by users first
- Making one critical table real-time at the cost of every other pipeline
- Backfilling silently and marking old data as freshly updated
Related concepts
Related examples
Authoritative source
What it is
Freshness answers one question at read time: how old is the newest fact in this table? Implementations instrument the pipeline to emit the age of each partition against its expected arrival, then alert when the age exceeds the agreed SLA. Because it tracks the data rather than the job, it catches silent stalls — a job that reports success but writes no new rows.
Why it matters
A fraud model scoring hour-old events, or a dashboard showing yesterday totals, can be confidently wrong. Freshness turns “the pipeline ran” into “the data is current enough for this decision”, which is the property consumers actually depend on. It also sets the right expectation: not every table needs to be seconds old, and stating the target prevents over-engineering.
How to manage it
Assign each dataset a freshness target tied to its decision, and measure the newest-record age in the pipeline. Alert on the data, not only on the job, so empty-but-successful runs are visible. Prefer incremental ingestion over full reloads to close the lag, and when a table inevitably falls behind, degrade gracefully — flag results as stale rather than serving them as current.