Use a Distributed Lock with Fencing

Acquire a lock with a token, renew it with a watchdog, and guard the critical section with a fencing token so a delayed holder cannot corrupt shared state.

~40 mindistributed-systemslockingredis

Task

Implement distributed mutual exclusion across processes with protection against lock expiry during a long operation.

When to use this

A lock expires while the holder is paused, a second process enters the critical section, and both write to shared state at the same time.

Prerequisites

A Redis-compatible store
A monotonic fencing source for writes

Steps

1. Acquire with SET NX PX and a random token, and store the token to identify ownership.

Expected output: Only one holder owns the lock at a time.

2. Renew the lock in the background while the critical section runs.

Expected output: The lock does not expire under a healthy holder.

3. Release only if the stored token matches, using a Lua script for atomicity.

Expected output: A late release cannot delete someone else lock.

4. Pass a fencing token to the guarded resource and reject writes with an older token.

redis-cli --eval release.lua lock-key , token

Expected output: A paused holder that resumes cannot overwrite newer state.

Troubleshooting

Problem

Lock not released on crash

Solution

Always use a TTL; never rely on manual release alone.

Problem

Clock issues and renewal

Solution

Renew based on a TTL well above operation time and use a watchdog thread with a safety margin.

Limitations

A distributed lock is not a transaction; correctness still depends on fencing at the resource.
Redis failover can violate mutual exclusion without consensus, so fencing tokens are mandatory for correctness.

Context

Implement distributed mutual exclusion across processes with protection against lock expiry during a long operation. A lock expires while the holder is paused, a second process enters the critical section, and both write to shared state at the same time.

Steps

Acquire with SET NX PX and a random token, and store the token to identify ownership. Expected: Only one holder owns the lock at a time.
Renew the lock in the background while the critical section runs. Expected: The lock does not expire under a healthy holder.
Release only if the stored token matches, using a Lua script for atomicity. Expected: A late release cannot delete someone else lock.
Pass a fencing token to the guarded resource and reject writes with an older token. Expected: A paused holder that resumes cannot overwrite newer state.

const token = crypto.randomUUID();
await redis.set(key, token, 'NX', 'PX', 30000);
// release atomically only if we still own it
await redis.eval(LUA_RELEASE, 1, key, token);

Verify

Only one holder owns the lock at a time.
The lock does not expire under a healthy holder.
A late release cannot delete someone else lock.
A paused holder that resumes cannot overwrite newer state.

Limits

A distributed lock is not a transaction; correctness still depends on fencing at the resource.
Redis failover can violate mutual exclusion without consensus, so fencing tokens are mandatory for correctness.