Add Fencing Tokens to Protected Writes

Attach a monotonically increasing token to every write and reject older tokens at storage so a paused lock holder cannot overwrite newer data.

~30 mindistributed-systemslockingdata-consistency

Task

Make a distributed lock safe against a holder that pauses past its lease and resumes after another process has taken over.

When to use this

A worker holds a lease, freezes during garbage collection, loses the lease, and wakes up believing it still owns the resource. Its delayed write overwrites the new owner's work.

Prerequisites

A lock or leadership lease with a monotonically increasing epoch or token
A storage layer that can compare the token atomically
All writes to the protected resource routed through the guarded path

Steps

1. Issue a token greater than every previous token when leadership is acquired, for example from a database sequence.

SELECT nextval('write_epoch') AS token;

Expected output: Every acquisition receives a strictly increasing token.

2. Store the highest accepted token and reject writes whose token is not greater.

UPDATE protected_rows
   SET value = $1, last_token = $2
 WHERE id = $3
   AND last_token < $2
RETURNING last_token;

Expected output: A write with an older token updates zero rows and can be reported as stale.

3. Test the pause scenario by acquiring a token, acquiring a newer one, then replaying the old write.

node --test fencing.test.mjs

Expected output: The old write is rejected while the new token's write succeeds.

Troubleshooting

Problem

Every write is rejected

Solution

The token is not strictly increasing or is compared as a string. Use an integer or a lexicographically sortable sequence.

Problem

Some code paths bypass the guard

Solution

Route all writes through one repository method; a direct UPDATE without the token check reopens the race.

Limitations

Fencing requires the protected resource to enforce the token; a log line or client-side check is not enough.
The token sequence itself must be durable and monotonic across failover.

A lock alone is not enough

A lease can expire while its holder is paused. The holder resumes and still believes it owns the lock. If it writes to shared storage, it can overwrite the new owner’s work. Fencing turns leadership into a monotonic token and makes storage reject stale writes.

term 7 token 7 -> write accepted
lease expires, term 8 token 8 -> write accepted
old holder resumes -> write token 7 -> rejected

Enforce at the resource

The guard must be in the same atomic operation as the write. WHERE last_token < $token in the update statement is atomic; a separate read-check-write is not. Return the new token or a row count so the caller can distinguish success from a stale rejection.

Test the pause

Simulate a pause by acquiring token 1, acquiring token 2, then issuing a write with token 1. Assert that the old write fails and the new value remains. Fencing is not a replacement for idempotency; a retried write with the current token should still be safe.

Verify the stale write

Expected outcome: the write with the older token updates zero rows, returns a distinguishable stale result, and leaves the newer value intact. Confirm with a test that acquires two tokens and replays the first after the second has written. Also verify that the token sequence survives a process restart; a per-process counter resets and breaks the ordering guarantee.