阅读中文版本

Pipeline Orchestration Dependency Workflow

Declare task dependencies explicitly so runs wait for real prerequisites instead of relying on cron timing.

Overview

Make pipeline run order depend on real dependencies rather than clock timing.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• An orchestrator such as Airflow or Dagster
• Task definitions
• Sensors or triggers

Procedure

1. Model tasks as a DAG

Represent each task and its true upstream dependencies as a directed graph.

Expected state: Dependencies reflect data flow, not schedule.

2. Replace time waits with sensors

Wait on upstream data availability instead of fixed delays.

Expected state: Tasks start when inputs exist, not when a clock says so.

3. Add failure propagation

Stop downstream tasks when an upstream fails.

Expected state: Failed runs do not cascade into corrupt outputs.

4. Verify with an induced delay

Delay an upstream task and confirm downstream waits appropriately.

Expected state: The DAG respects actual completion, not timing.

# wait on upstream data rather than a fixed delay
task_downstream >> task_upstream
# or use a sensor on the upstream dataset

Verification

Confirm delaying an upstream task delays downstream start rather than causing stale reads.

Decision points

Schedule-based or dependency-based?

• Dependency-based
• Clock-based waits

Recommendation: Use dependency-based ordering; clock waits fail when upstream is late.

Fail fast or continue on partial failure?

• Stop downstream
• Continue where safe

Recommendation: Stop downstream of a failure unless the task explicitly tolerates partial input.

Common failures

• Clock waits read stale upstream data.
• Hidden dependencies break when schedules shift.
• Failures cascade silently into corrupt outputs.