Enable Terraform State Locking and Unlock Safely

Lock remote state with a conditional write or DynamoDB table, and force-unlock only after confirming no apply is still running.

~25 minterraformstatereliability

Task

Prevent two pipelines from applying the same state at once, and recover from a stale lock without corrupting state.

When to use this

A cancelled CI job leaves a lock behind. The next deploy fails with a lock error, and someone force-unlocks while a manual apply is still running.

Prerequisites

An S3 or equivalent remote backend
Permissions to create a lock table or use conditional writes
Terraform 1.10 or newer for lockfile-based locking

Steps

1. Configure the backend with locking and encryption.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "payments/prod.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "acme-tfstate-locks"
    encrypt        = true
  }
}

Expected output: terraform init succeeds and every plan or apply acquires the state lock.

2. Prove the lock blocks a second writer.

terraform plan -lock-timeout=0 & terraform plan -lock-timeout=0; wait

Expected output: The second plan fails with an acquiring-lock error that includes a lock ID and creation time.

3. Inspect the lock holder before deciding to unlock.

aws dynamodb get-item --table-name acme-tfstate-locks --key '{"LockID":{"S":"acme-tfstate/payments/prod.tfstate"}}'

Expected output: The item shows the lock ID, owner, and creation time; confirm that CI run is dead.

4. Force-unlock only the verified lock ID.

terraform force-unlock "$LOCK_ID"

Expected output: The lock is removed and the next plan succeeds; no apply was running at the time.

Troubleshooting

Problem

The lock survives a cancelled CI job

Solution

Check the lock metadata and the run status first; wait for the job to finish or cancel it, then unlock the exact lock ID.

Problem

DynamoDB table not found

Solution

Create a table with a string partition key named LockID, or use use_lockfile with S3 conditional writes on Terraform 1.10 and newer.

Limitations

Locking protects state writes, not cloud APIs; provider-side conflicts still need unique resource names.
force-unlock bypasses the safety check and can create two concurrent writers if misused.

Why locking exists

Terraform reads state, computes a diff, calls provider APIs, and writes state back. Two runs doing this against the same state can interleave and produce a state file that describes neither reality. Locking serializes that sequence by having the backend record an owner while a run is active. S3 with DynamoDB has been the standard pattern; newer Terraform versions can use S3 conditional writes with use_lockfile = true, which removes the extra table. Either way, the lock is only useful if writers respect it, so every pipeline and every human must use the same backend configuration.

backend "s3" {
  bucket         = "acme-tfstate"
  key            = "payments/prod.tfstate"
  region         = "eu-west-1"
  dynamodb_table = "acme-tfstate-locks"
  encrypt        = true
}

Unlocking is an incident decision

A stale lock is usually a cancelled job, but it can also be a live apply running from a laptop. Before unlocking, read the lock metadata: the ID, the owner, and when it was created. Check the CI run and any current on-call activity. Unlock the specific ID rather than the whole table, and only after you are certain no writer is active. If the apply did run to completion, run terraform refresh or a read-only plan immediately, because a state write may have been interrupted mid-way.

Practice the recovery

Run the drill on a non-production stack: start a plan, kill the process, and walk through inspect, unlock, and re-plan. Record the commands in the runbook. Teams that rehearse this recover in minutes; teams that improvise during a release either wait for an expired lock or corrupt state by unlocking a live apply.