Checkpoint-Restart Recovery
Checkpoint-restart periodically persists the progress of a long-running job so that a failure resumes from the last checkpoint rather than restarting from the beginning.
Definition
Checkpoint-restart recovery is the technique where a long-running process periodically writes its progress — committed offsets, processed keys, intermediate state — to durable storage, so that on failure it resumes from the last checkpoint rather than repeating all prior work, bounding the re-work to the checkpoint interval.
Context and scope
Applied in batch ETL, streaming consumers, long-running ML training, and any job whose restart cost exceeds the checkpointing cost.
Common misconceptions
- Checkpointing is just saving state — it is saving state at a point from which the computation can be deterministically resumed, which is stricter than a save
- Exactly-once processing requires checkpoints — it requires checkpoints plus idempotent application of the resumed work; the checkpoint alone is not enough
Anti-patterns
- Checkpointing so frequently that the checkpoint write cost dominates the processing cost
- Checkpointing without making the computation deterministic, so resumed work produces different results than the original run
Related concepts
Authoritative source
https://spark.apache.org/docs/latest/streaming/programming-guide.html
What it is
Checkpoint-restart recovery is the technique where a long-running process periodically writes its progress — committed offsets, processed keys, intermediate state — to durable storage, so that on failure it resumes from the last checkpoint rather than repeating all prior work, bounding the re-work to the checkpoint interval. Applied in batch ETL, streaming consumers, long-running ML training, and any job whose restart cost exceeds the checkpointing cost. The data model and the access patterns are co-designed: a schema that is correct in the relational sense can be catastrophic for the queries the workload actually issues.
How it works
Indexing decisions follow the query shapes, not the column importance: a covering index turns a table scan into an index-only scan, but every index adds write amplification and storage cost. Partitioning aligns with the access dimension so a query touches one partition, not all. Transaction scope is minimized to hold locks briefly; long transactions are the root of contention and deadlock.
When to apply and pitfalls
Apply when data volume or query latency has a concrete business cost. The common pitfall is indexing for the current workload and ignoring the access patterns that growth will introduce — the index that helps today can serialize writes tomorrow. Another is treating the database as a black box, missing that the optimizer’s plan choice depends on statistics that drift as data changes.