Implement Leader Election with Leases

Elect a single leader using a renewable lease, let followers detect expiry, and make steps idempotent so a brief dual-leader window is harmless.

~45 mindistributed-systemsschedulingreliability

Task

Add leader election so scheduled work runs on exactly one instance without a fixed coordinator.

When to use this

A cron job runs on every replica, causing duplicate reports, duplicate notifications, and race conditions on shared state.

Prerequisites

A consensus or lease store such as etcd or a database
Idempotent work units

Steps

1. Store a lease with a holder id and a TTL in the coordination store.

Expected output: At most one instance sees itself as leader.

2. Refresh the lease on an interval shorter than the TTL while leader.

Expected output: A healthy leader retains leadership indefinitely.

3. Have followers watch the key and attempt to acquire when it disappears.

Expected output: A new leader emerges after the old one expires.

4. Make every work unit idempotent so overlapping windows do not double-apply.

etcdctl lease grant 15 && etcdctl put leader --lease=<id>

Expected output: A transient dual-leader period causes no corruption.

Troubleshooting

Problem

Split brain on long pauses

Solution

Keep TTL short relative to work duration and make work idempotent so a stalled leader cannot corrupt state.

Problem

Leadership flapping

Solution

Set backoff between acquisition attempts and a minimum stable period before a follower retries.

Limitations

Leases tolerate brief dual leadership, so they are safe only for idempotent work.
They do not provide strong consistency for arbitrary critical sections; use consensus primitives for those.

Context

Add leader election so scheduled work runs on exactly one instance without a fixed coordinator. A cron job runs on every replica, causing duplicate reports, duplicate notifications, and race conditions on shared state.

Steps

Store a lease with a holder id and a TTL in the coordination store. Expected: At most one instance sees itself as leader.
Refresh the lease on an interval shorter than the TTL while leader. Expected: A healthy leader retains leadership indefinitely.
Have followers watch the key and attempt to acquire when it disappears. Expected: A new leader emerges after the old one expires.
Make every work unit idempotent so overlapping windows do not double-apply. Expected: A transient dual-leader period causes no corruption.

if (await store.acquire(leaseKey, instanceId, ttl)) {
  startLeaderWork();
  setInterval(() => store.renew(leaseKey, instanceId, ttl), (ttl * 1000) / 3);
}

Verify

At most one instance sees itself as leader.
A healthy leader retains leadership indefinitely.
A new leader emerges after the old one expires.
A transient dual-leader period causes no corruption.

Limits

Leases tolerate brief dual leadership, so they are safe only for idempotent work.
They do not provide strong consistency for arbitrary critical sections; use consensus primitives for those.