阅读中文版本

Data Lineage Capture Workflow

Record how datasets are derived from sources so impact analysis and debugging can trace upstream and downstream.

Overview

Capture end-to-end data lineage to support impact analysis and debugging.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• A lineage tool or metadata store
• Pipeline definitions
• A naming convention

Procedure

1. Emit lineage from jobs

Have each job report its inputs and outputs to the lineage store.

Expected state: Every job records its source-to-target edges.

2. Model datasets consistently

Use consistent dataset names so lineage resolves across teams.

Expected state: Lineage connects across pipelines.

3. Query impact when changing a schema

Use lineage to find every downstream dataset before changing an upstream one.

Expected state: Changes are assessed against all consumers.

4. Debug with lineage

Trace upstream from a suspect dataset to find the source of bad data.

Expected state: Root of a data issue is found quickly.

# emit lineage from a job
datahub emit-lineage --in raw.orders --out curated.orders_enriched

Verification

Confirm lineage resolves across pipelines and that impact and root-cause queries return the expected datasets.

Decision points

Manual or automatic lineage?

• Automatic from jobs
• Manually documented

Recommendation: Automatic emission stays accurate; manual docs drift.

Column-level or table-level?

• Column-level for sensitive data
• Table-level first

Recommendation: Start table-level and add column-level where PII or compliance demands it.

Common failures

• Manual lineage drifts from reality.
• Inconsistent naming breaks the graph.
• No lineage makes impact analysis guesswork.