Configure Log Retention and Archival

Classify logs by purpose and sensitivity, set hot/warm/cold retention, archive to object storage, and prove deletion without losing audit evidence.

intermediate~45 minloggingprivacyoperations

Objective

Reduce log storage cost and privacy exposure while keeping the logs needed for debugging, security, and compliance.

Environment

A log pipeline with index or bucket lifecycle support
A data classification for application, access, audit, and debug logs
Object storage with encryption and access controls
Retention requirements from security and legal

Steps

1. Classify logs

Separate application logs, access logs, audit events, and debug traces. Mark which contain personal data or secrets and which have legal retention.

Expected state: Every log stream has a purpose, owner, and retention rule.

2. Define hot/warm/cold tiers

Keep recent logs searchable, move older logs to cheaper storage, and delete when the retention period ends.

Expected state: The lifecycle policy matches the query and compliance needs.

3. Redact before storage

Remove or hash secrets, tokens, and unnecessary personal data at ingestion or before archival.

Expected state: Archived logs do not contain credentials or unapproved personal data.

4. Verify deletion and access

Test that expired logs are deleted, archived logs are encrypted and access-controlled, and legal holds override deletion.

Expected state: Retention is enforceable and auditable.

Decision points

Should audit logs use the same retention as application logs?

Separate, longer retention for audit
One policy for all logs

Keep audit logs separate with stronger access control and a retention period driven by compliance.

Should logs be archived or deleted?

Archive to encrypted object storage
Delete at the end of the hot period

Archive only if there is a documented need; deleting reduces cost and privacy risk.

Expected outcome

Logs are retained only as long as needed, searchable when needed, and deleted or archived under a documented policy.

Verification

Query the index and object storage for a record older than the retention period, confirm it is gone or archived, and verify access controls and legal-hold exceptions.

Cleanup

Update the log catalog, remove obsolete dashboards, and document the retention schedule and owners.

Logs are data with a lifecycle

Application logs, access logs, audit events, and debug traces have different value and risk. An audit event may need years of retention and strict access; a debug log may be useful for days. Treating all logs the same either costs too much or deletes evidence too early.

lifecycle:
  hot: 7d
  warm: 30d
  cold: 365d
  delete_after: 365d

Redact before archiving

Logs frequently contain tokens, cookies, and personal data. Redact at ingestion or before archival. Archiving a secret for a year does not make it safe; it extends the exposure. Apply the same access controls to archives as to the live index, and encrypt at rest.

Prove the policy works

Write a test record with a known timestamp, wait for or simulate the lifecycle transition, and verify it is deleted or archived as expected. Check that legal holds prevent deletion and that expired data is actually removed from every replica and index. A policy configured but never verified is a guess.