Data Lineage
Data lineage records how datasets and columns are derived, so teams can trace errors, assess change impact, and prove where regulated data came from.
Definition
Data lineage is the recorded graph of data movement and transformation: which sources, jobs, tables, and columns produced a given dataset. Table-level lineage shows dataset dependencies; column-level lineage traces a specific value through joins, filters, and aggregations.
Context and scope
Lineage is essential when a dashboard number looks wrong, when a schema change must be assessed, or when an auditor asks which source systems contributed to a report. It is also a prerequisite for trustworthy impact analysis.
Common misconceptions
- Lineage is a static architecture diagram that can be drawn once.
- Table-level dependencies are enough to debug a wrong column value.
Anti-patterns
- Collecting lineage only from manually maintained documentation that drifts from the pipeline.
- Storing lineage without timestamps or job-run identifiers, so it cannot explain a specific incident.
Related concepts
Related examples
Authoritative source
How it works
Lineage is captured from three sources: SQL parsing, orchestrator metadata, and runtime events. A SQL parser extracts input and output tables and, when it can resolve aliases, column mappings. An orchestrator knows which job ran, with which parameters, at what time. Runtime integrations emit start and finish events with input and output datasets, forming a run-level graph.
The graph is usually stored as nodes (datasets, jobs, columns) and edges (reads, writes, derives). Column-level lineage is harder because expressions, unions, and user-defined functions obscure the mapping. Many systems offer table-level accuracy everywhere and column-level accuracy for simple transformations.
What it is used for
The first use is debugging: a bad number can be traced upstream to the job or source that introduced it, and downstream consumers can be identified before a fix is published. The second is impact analysis: before dropping a column, find every report and model that reads it. The third is governance: prove that personal data flowed only through approved systems and was deleted where required.
Trade-offs
Lineage requires instrumentation across many engines and often struggles with dynamic SQL, temporary tables, and files whose paths are computed at runtime. Storage and UI costs grow with graph size. Stale lineage is worse than none because it creates false confidence. Column-level precision is expensive and should be prioritized for regulated or high-value data.
Example and misconception
A revenue dashboard drops 3 percent. Table-level lineage shows the dashboard depends on fct_revenue, but not which source column changed. Column-level lineage traces the drop to a currency conversion join that lost rows after a country code rename. Without that edge, the team would inspect every upstream table.
The misconception is that lineage is a one-time diagram. Real lineage is event-based, versioned with the pipeline, and only useful when it reflects the run that produced the data.