Data Quality Monitoring
Data quality monitoring continuously checks the dimensions — completeness, consistency, timeliness, accuracy — that the data contract specifies, so a quality regression is detected before it.
Definition
Data quality monitoring is the continuous measurement of data against the dimensions the contract specifies — completeness (no nulls where non-null is required), consistency (referential integrity), timeliness (freshness within SLA), accuracy (value in expected range) — so that a quality regression raises an alert before the bad data propagates to downstream consumers.
Context and scope
Applied in data pipelines, analytics, and any system where decisions are made from the data and a quality regression has a business cost.
Common misconceptions
- Quality monitoring is just data validation — validation is one-time at ingest; monitoring is continuous, so a quality regression after ingest is caught
- Passing validation means the data is good — validation checks the schema; quality monitoring checks the statistical properties, which validation cannot
Anti-patterns
- Monitoring only at ingest, so a quality regression that develops in the stored data (decay, corruption) is not caught
- Alerting on every quality metric without thresholds, so the team drowns in alerts and ignores the real regression
Related concepts
Authoritative source
What it is
Data quality monitoring is the continuous measurement of data against the dimensions the contract specifies — completeness (no nulls where non-null is required), consistency (referential integrity), timeliness (freshness within SLA), accuracy (value in expected range) — so that a quality regression raises an alert before the bad data propagates to downstream consumers. Applied in data pipelines, analytics, and any system where decisions are made from the data and a quality regression has a business cost. The data is the asset; the pipeline is the liability — it exists only to move and transform data correctly, on time, with a lineage that survives audit.
How it works
Schema is a contract enforced at ingestion, not discovered at query time: bad records are quarantined with their violation, not silently coerced. Transformations are deterministic and idempotent so re-runs repair rather than duplicate. Partitioning by event time (not processing time) makes late data correctable rather than lost. Lineage records every transformation so the path from source to dashboard is reconstructable.
When to apply and pitfalls
Apply when decisions are made from the data and a wrong decision has a cost. The pitfall is treating the pipeline as a one-way function and never validating its output against the source — silent drops and double-counts accumulate until a dashboard disagrees with the ledger. Another is optimizing for throughput at the cost of correctness, dropping the validation that would have caught the corruption.