Data Lineage Documentation Workflow
Capture field-level lineage automatically from pipeline code, publish it as a queryable graph, and use it for impact analysis before changes.
Objective
Maintain lineage that answers which downstream assets break when a field changes, generated from the pipelines themselves rather than maintained by hand.
Environment
- Pipelines expressed in parseable SQL or orchestration code (dbt, Spark, Airflow)
- A lineage tool or graph store
- A convention that every asset is registered with an owner
Steps
1. Generate lineage from code, not wikis
Parse pipeline definitions to extract table and column dependencies automatically, rejecting hand-maintained lineage diagrams that rot.
Expected state: The lineage graph regenerates on every pipeline deploy with no manual edits.
2. Register owners per asset
Attach a named owner to every table, dashboard, and model in the graph, so impact analysis produces contacts, not just names.
Expected state: Every node in the graph has an owner resolvable to a real person.
3. Publish as a queryable graph
Expose lineage through an API or UI that answers both directions: what depends on this field, and where does this number come from.
Expected state: Both upstream and downstream queries return in seconds from the same graph.
4. Run impact analysis before changes
Make schema or logic changes cite the lineage impact list: affected tables, dashboards, and owners, who ack before the change merges.
Expected state: Change requests include a machine-generated impact list with owner acks.
5. Audit lineage coverage
Track what percentage of production assets have generated lineage and owners, driving registration of the stragglers.
Expected state: Coverage is a reported metric, and unregistered assets are visible by their absence.
Expected outcome
A lineage graph generated from pipeline code that answers dependency questions in seconds and gates changes with owner acks instead of tribal memory.
Verification
A field rename impact query returns the complete affected set including a dashboard three hops away; the coverage metric is above the team threshold.
Lineage that rots is worse than none
A hand-drawn lineage diagram confidently wrong is more dangerous than no diagram: it answers impact questions with fiction. Generating lineage from the pipeline code itself keeps it truthful by construction, regenerating on every deploy.
question: rename orders.total_cents -> total_amount_cents
lineage impact (generated, 0.4s):
tables (3 hops):
wh.fct_orders (direct)
wh.agg_daily_revenue (selects total_cents)
dashboards:
revenue_v3 (uses agg_daily_revenue)
finance_exec (uses agg_daily_revenue)
owners acked: analytics-lead, finance-data-eng
coverage: 96% assets registered (4 orphans listed)Generated beats maintained
Every manual diagram diverges from reality the week after it is drawn, and the divergence is invisible. Parsing the actual SQL or DAG code regenerates truth on each deploy, so the graph answers what is rather than what someone remembered.
Impact analysis with owners turns theory into contacts
Knowing that three dashboards break is trivia; knowing that finance_exec breaks and its owner is wei@team, who has acked the change, is governance. The lineage graph earns its keep at the moment of change, not as an archive.