Pipeline Data Quality Gates Workflow
Insert assertion-based gates between pipeline stages that check completeness, validity, and referential integrity, quarantining bad batches.
Objective
Stop corrupted or incomplete data from propagating downstream by failing fast at the stage boundary where the problem entered.
Environment
- A pipeline with identifiable stage boundaries
- An assertion framework (Great Expectations, dbt tests, or custom SQL checks)
- A quarantine location for failed batches
Steps
1. Define invariants per stage boundary
Write assertions for each stage output: not-null on keys, value ranges, uniqueness, referential integrity, and row-count bands.
Expected state: Each stage boundary has a named, versioned set of assertions.
2. Fail closed at the gate
Configure gates so a failed assertion stops the batch from proceeding, quarantining it with the failing assertion and a sample of offending rows.
Expected state: Bad data lands in quarantine with evidence; downstream never sees it.
3. Quarantine with diagnosis context
Store the failed batch with its assertion results, input lineage, and a row sample so triage can happen without re-running the pipeline.
Expected state: A quarantined batch is self-describing: what failed, which rows, from which input.
4. Route failures by invariant class
Key or referential failures page the pipeline owner immediately; range or count drift goes to the morning quality digest.
Expected state: Two routing classes exist with owners, mirroring severity in alerting.
5. Replay after fix with full re-check
Once the cause is fixed, replay the quarantined batch through the same gates, verifying every assertion passes before it merges downstream.
Expected state: Replayed batches traverse the identical gate set; passing is earned, not assumed.
Expected outcome
Pipelines where corrupted batches stop at the boundary they corrupted, quarantine with diagnosis evidence, and re-enter only through the same assertions.
Verification
An injected bad batch (null key) stops at the first gate with the sample rows in quarantine; downstream tables show no trace of it; replay after fix passes all gates.
Bad data downstream is a choice the pipeline made
Once corrupted rows reach the warehouse and reports, the fix becomes an archaeology project. Assertion gates at stage boundaries keep the problem where it entered, quarantine it with evidence, and require the full gate pass to re-enter.
gate: load_orders (stage boundary)
assertions:
order_id not-null, unique PASS
status in {pending,paid,...} PASS
total_cents > 0 AND < 10_000_000 FAIL (17 rows = 0)
customer_id 97% referential match FAIL (412 orphans)
result: BATCH QUARANTINED
reason: 2 failed assertions + row samples
lineage: extract.run 2026-10-22-00-31
routing: referential failure -> page pipeline owner
replay (after fix): all assertions PASS -> mergeFail closed, quarantine loud
A gate that logs and continues is a suggestion, and suggestions get ignored during incidents. Failing closed with a quarantine that carries the failing assertion, offending row samples, and lineage turns each failure into a self-contained diagnosis package.
Replay earns its passage
The temptation after a fix is to hand-merge the quarantined data to catch up. Forcing the replay through the identical gates — including the assertions it originally failed — guarantees the fix actually addressed the cause rather than the symptom.