Split Brain

Split brain occurs when two parts of a distributed system both believe they are authoritative, accepting divergent writes that cannot be merged safely.

distributed-systems

Definition

Split brain is a failure mode in which two nodes or partitions independently believe they are the active primary or leader. Both accept writes, so the system develops divergent histories. When the partition heals, one side's writes may be lost or must be reconciled. Split brain is prevented with quorum, fencing, and a single authoritative decision path, not by hoping the network stays healthy.

Context and scope

Split brain appears in database failover, cluster managers, distributed locks, and any system with a primary/replica topology. It is especially dangerous because each side appears healthy from its own perspective.

Common misconceptions

Automatic failover prevents split brain — a failed health check can trigger a second primary while the first still serves
A lease timeout guarantees the old primary stops — a paused process can resume and continue writing
Split brain always causes visible errors — both sides may serve successfully until reconciliation

Anti-patterns

Promoting a replica without fencing the old primary
Allowing a minority partition to accept writes
Manual failover without confirming the old primary is stopped or isolated

Related concepts

Authoritative source

https://www.postgresql.org/docs/current/warm-standby.html

Two primaries, two histories

A network partition separates the primary from the replicas and the failover controller. The controller promotes a replica because the primary appears dead. The original primary is still running and serving clients on the other side. Both accept writes. When the network returns, the system has two divergent histories and no automatic way to merge them without losing data or violating invariants.

Partition A: old primary  -> accepts writes
Partition B: new primary  -> accepts writes
             (no communication between A and B)

Prevent the second writer

The strongest defense is to ensure that only a side with quorum can accept writes. A majority quorum cannot exist on both sides of a partition, so at most one side can elect a leader. The minority side must reject writes or become read-only. This is a deliberate availability trade-off: the minority is unavailable for writes rather than divergent.

Fencing adds a second layer. Every write carries a monotonically increasing token or epoch. Storage rejects writes with an older token, so an old primary that resumes cannot overwrite the new primary’s data. This protects against pauses, delayed health checks, and human error during failover.

Failover is a controlled procedure

Automatic failover must confirm that the old primary is stopped, isolated, or fenced before the new primary accepts writes. A health check that only tests the process is insufficient; it may fail while the process still holds resources and clients. Use the platform’s fencing mechanism, and monitor for two nodes claiming the primary role.

After healing, do not simply reconnect both sides. Inspect the timelines, identify divergent writes, and follow a documented reconciliation or restore procedure. Preserve the old primary’s data for analysis. A split-brain incident is a data-integrity incident, even if the service stayed up.

When to use this lens

Use split-brain reasoning when designing failover, leader election, or multi-region writes. The advantage is explicit safety rules; the drawback is reduced write availability during partitions. Compared to a single-node database, a highly available topology adds failure modes that must be tested, not assumed away.