Leader Election

Leader election chooses one node to perform singleton work, using leases or consensus plus fencing tokens so a paused old leader cannot keep writing.

distributed-systems

Definition

Leader election is the process by which a group of nodes agrees that one member is responsible for a singleton task, such as accepting writes, scheduling jobs, or coordinating a partition. A lease or consensus protocol grants leadership for a bounded term. Correctness depends on fencing: every write carries a monotonically increasing token so storage can reject a stale leader that has not yet noticed it lost the lease.

Context and scope

Election is needed whenever two workers doing the same job would cause duplicate side effects or conflicting writes. Examples include a scheduler, a queue partition owner, and a primary database.

Common misconceptions

A lease guarantees the old leader stops immediately — a paused process may resume and still believe it is leader
Leader election alone prevents split brain — without fencing or quorum, two nodes can act as leader during a partition
The fastest node should always lead — stability and fencing matter more than raw latency

Anti-patterns

Using a database row as a lock without a fencing token or an expiry-aware compare-and-set
Holding a leadership lease longer than the work can safely overrun
Electing a leader without a quorum, so two sides of a partition each elect one

Related concepts

Authoritative source

https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

Leadership is a lease, not a crown

A node becomes leader for a term by acquiring a lease or winning a consensus round. While the lease is valid, it may perform singleton work. The hard case is a leader that pauses—GC, VM freeze, network delay—longer than the lease. Another node acquires leadership, but the paused process wakes up still believing it leads. If it writes to shared storage, the system has two leaders.

Fencing solves this. Each leadership term has a monotonically increasing token. Every write includes the token, and the storage rejects a write whose token is lower than the highest it has seen. The old leader may still think it is in charge, but its writes fail.

term 7 leader A -> write(token=7) -> accepted
A pauses, lease expires
term 8 leader B -> write(token=8) -> accepted
A resumes -> write(token=7) -> rejected by storage

Choose the coordination layer

A consensus-backed store such as etcd, Consul, or ZooKeeper provides leases and watch notifications. A relational database can implement election with a compare-and-set row and an expiry, but the application must handle clock skew and fencing. Do not use a plain SELECT ... FOR UPDATE held for the duration of long work; it blocks, times out, and does not survive a process pause cleanly.

Design for handoff

The new leader should wait for the old leader’s lease to expire and for any in-flight work to settle before taking over. Work should be idempotent or fenced so a duplicate attempt cannot corrupt state. Graceful shutdown should release leadership deliberately; a crash should rely on lease expiry. Monitor election churn: frequent leader changes usually indicate network instability, overload, or leases that are too short for the work.

When to use it

Use leader election when exactly one actor may perform a task and duplicate execution is harmful. The advantage is a single writer and a clear ownership model; the drawback is a dependency on a coordination system and a failover delay. Compared to partitioning work by key, election is simpler for a global singleton but harder to make highly available.