Security Incident Response Runbook

Contain a suspected credential compromise, preserve evidence, rotate affected secrets, eradicate persistence, and recover with verification and clear communication.

advanced~120 minsecurityincident-responseoperations

Objective

Limit damage from a security incident, preserve evidence, restore trusted access, and produce a factual timeline for notification and follow-up.

Environment

An on-call security or platform owner with authority to revoke access
Centralized audit logs for identity, cloud, and application events
A secret inventory that maps credentials to services and owners
An out-of-band communication channel

Steps

1. Declare and scope

Assign an incident lead, record the time, and state the suspected scope without speculating publicly. Open a private channel and a timeline document.

Expected state: One owner, one channel, and a timestamped record exist.

2. Contain without destroying evidence

Revoke the suspected session or key, isolate the affected workload, and preserve logs and disk snapshots before rebuilding.

Expected state: The actor loses access while evidence remains available for analysis.

3. Rotate and verify credentials

Rotate every credential reachable from the compromised identity, including tokens, database passwords, and CI secrets. Verify old credentials fail.

Expected state: Old credentials are rejected and dependent services use the new values.

4. Eradicate persistence

Inspect scheduled jobs, deploy keys, OAuth grants, webhooks, and IAM roles for changes made during the incident; remove unauthorized persistence.

Expected state: No unknown access path remains.

5. Recover and monitor

Restore from a trusted build or snapshot, re-enable traffic gradually, and monitor for recurrence with heightened alerts.

Expected state: Service is restored from known-good artifacts and the actor's access does not return.

6. Notify and review

Follow legal and contractual notification requirements, then run a blameless review with concrete prevention actions and owners.

Expected state: Required notifications are sent and remediation work is tracked.

Decision points

Should the affected service be taken offline immediately?

Isolate first, then investigate
Keep serving while investigating to avoid business impact

Isolate when active compromise is suspected. Containment takes priority over availability for confirmed intrusions.

Who approves credential rotation during an incident?

Incident lead with security on-call
A full change-approval board

Pre-authorize emergency rotation in the runbook; a board is too slow during active compromise.

Expected outcome

The compromised access is removed, evidence is preserved, the service is restored from trusted artifacts, and the incident produces owned prevention work.

Verification

Confirm revoked credentials fail, audit logs show no further actor activity, restored builds match signed artifacts, and every remediation item has an owner and due date.

Cleanup

Close temporary network blocks after the threat is gone, archive evidence under retention policy, and update the runbook with what actually happened.

Contain first, but keep the evidence

Deleting the compromised workload feels decisive, but it also destroys the logs and state needed to understand scope. Snapshot first, then isolate. Preserve identity logs, network flow logs, application logs, and the build artifact that was running. Record who performed each action and when.

# Example: preserve the workload's recent logs before replacement
kubectl logs deploy/api --since=6h > incident-api-logs.txt
kubectl get events --sort-by=.lastTimestamp > incident-events.txt

The exact commands depend on the platform. The principle is platform-independent: capture evidence before destructive remediation.

Rotate reachable credentials

A compromised CI token can usually read every secret the pipeline can read. Map the blast radius from the secret inventory, then rotate in dependency order. Revoke first, verify the old value is rejected, then distribute the new value through the secret manager. Do not paste new secrets into the incident channel.

Look for persistence

Attackers rarely stop at one key. Check for new IAM users, deploy keys, OAuth applications, webhooks, scheduled tasks, and role trust policies. Compare the current configuration with a known-good baseline or infrastructure-as-code state. Remove unauthorized changes and record them as evidence.

Communicate in facts

Separate confirmed facts from hypotheses. Report the affected systems, the time window, the containment status, and the next update time. Do not speculate about attribution or customer impact before validation. After recovery, run a blameless review focused on detection time, containment time, and the controls that failed.