Failover
Failover shifts traffic or workload to a standby when the primary fails, and its reliability depends on detection, state, and regular rehearsal.
Definition
Failover is the process of transferring service from a failed or unhealthy component to a redundant one, automatically or through an operator.
Context and scope
Used for databases, load balancers, applications, and regions, especially where recovery time objectives are measured in minutes.
Common misconceptions
- Failover is instant and invisible to users
- A standby that starts is a standby that works
- Failover is complete once traffic moves
Anti-patterns
- Configuring automatic failover without split-brain protection
- Never rehearsing failover until a real outage
- Ignoring failback and data reconciliation
Related concepts
Authoritative source
https://learn.microsoft.com/en-us/azure/architecture/reliability/architect
How failover works
Failover moves service from a failed primary to a redundant secondary. The sequence sounds simple but contains several distinct problems: detecting that the primary is unhealthy, deciding who is allowed to promote the secondary, redirecting traffic, and confirming that the new primary can serve requests. Each step can fail independently. A health check that cannot distinguish slow from dead triggers unnecessary failovers; a promotion without quorum creates two primaries.
Detection and quorum
Detection needs multiple signals and a tolerance for transient errors. A single failed health check should not move a database. Distributed systems also need a rule for preventing split-brain: a majority quorum, a lease, or an external arbiter. Automatic failover is attractive because it is fast, but it can amplify a network partition into two writable copies. Compared to automatic failover, manual failover is slower but allows judgment; many teams use automatic detection with a human approval gate for stateful systems.
State and data
Stateless failover is mostly routing. Stateful failover must account for replication lag and lost writes. Define the recovery point objective, monitor lag continuously, and decide whether the new primary may accept writes before catching up. DNS-based failover adds TTL delays and client caching; connection draining avoids cutting active requests.
Rehearsal
Test failover on a schedule, including the detection path, promotion, traffic shift, and application behavior during the gap. Measure actual recovery time against the target. For example, a standby that takes four minutes to warm a cache may meet the database target but violate the user-facing objective. Record how long each step took.
Failback
The job is not done when traffic moves. Failback needs a controlled plan, data reconciliation, and a check that the old primary is healthy and caught up. Skipping failback leaves the system running on a degraded standby indefinitely.
Limits
Failover handles component loss, not correlated outages or data corruption. A replica that receives the same bad migration fails just as fast as the primary.