Data Retention
Data retention defines how long each class of data is kept and how it is destroyed. Clear schedules reduce storage cost, legal exposure, and privacy risk.
Definition
Data retention is the practice of defining, enforcing, and documenting how long each category of data is stored in each system and how it is deleted or anonymized once that period ends, including copies in logs, backups, caches, and warehouses.
Context and scope
Retention is shaped by legal obligations, contractual terms, and business needs. It applies to primary stores, logs, backups, caches, analytics warehouses, and derived exports, each with an independent lifecycle and owner.
Common misconceptions
- Deleting a row in the primary database deletes the data
- Backups may keep data forever because they exist only for recovery
- Longer retention is always safer for compliance
Anti-patterns
- Publishing a retention policy without an automated deletion job
- Keeping production data indefinitely in analytics or staging copies
- Deleting records without checking legal holds and audit requirements
Related concepts
Related examples
Authoritative source
How retention works
Classify data by purpose and legal obligation, set a period per class, automate enforcement, and verify deletion. Enforcement mechanisms include TTL indexes, scheduled jobs, partition drops, object-storage lifecycle rules, and crypto-shredding for encrypted blobs.
-- Keep 13 monthly partitions; dropping one is cheaper than row-by-row deletes
DROP TABLE events_2024_09;Backups need their own schedule: a 30-day deletion may live on in a 90-day backup until that backup expires, so document the maximum lag. Derived copies such as search indexes, caches, warehouses, and exports must be included or they become shadow archives.
When to use it
Define retention before a store receives production data, and revisit when adding analytics, PII, or a new region. Erasure requests and legal holds require both a written policy and a tested procedure. Compared with access control, retention limits exposure over time rather than at read time.
Trade-offs
Short retention lowers storage cost and privacy exposure but removes history that incident response or finance may need. Automated deletion can break features that silently depend on old rows. Legal holds must override deletion. Deletion in immutable systems is often logical first and physical later, which affects promised deadlines.
Concrete example
A retention matrix lists orders at seven years in monthly partitions, application logs at 30 days through an index TTL, analytics events at 14 months with partition expiration, and support attachments at two years with object lifecycle rules. Each row names an owner and a verification query that proves old data is gone.
Misconceptions and related concepts
Deleting a primary row does not erase backups, caches, or replicas, and longer retention is not automatically safer because it increases breach impact and legal exposure. Related concepts include PII, Backup and Restore, Data Residency, and Audit Logging.