Clock Skew
Clock skew is the difference between clocks on different machines; it breaks timestamp ordering, lease safety, and last-write-wins conflict resolution.
Definition
Clock skew is the deviation between a host's clock and a reference time, or between two hosts. It arises from oscillator drift, virtualization, NTP adjustment, and suspend/resume. Skew can make an event that happened later appear earlier, so wall-clock timestamps are unsafe for ordering distributed writes without additional mechanisms.
Context and scope
Distributed systems routinely compare timestamps for conflict resolution, cache expiry, token validation, and leases. A few hundred milliseconds of skew can turn a last-write-wins policy into silent data loss or let an expired lease remain valid.
Common misconceptions
- NTP synchronization makes clocks exact, so timestamp order equals causal order.
- Monotonic clocks can be compared across machines to order events.
Anti-patterns
- Using wall-clock time alone to decide which of two concurrent writes wins.
- Deriving lease expiry only from a local clock without a safety margin or fencing token.
Related concepts
Related examples
Authoritative source
How it works
Every machine has a quartz oscillator that drifts with temperature and age. NTP or PTP periodically estimates the offset and slew or steps the clock. Virtual machines can pause, and a resumed VM may jump forward. Even well-managed fleets see milliseconds of skew; misconfigured hosts can be seconds or minutes off.
Wall-clock time answers “what time is it here”, not “which event happened first across machines”. Monotonic clocks are immune to jumps and are correct for measuring durations on one host, but their epochs are arbitrary and cannot be compared across hosts. Logical clocks, such as Lamport timestamps and vector clocks, capture causality without relying on physical time.
Where it causes damage
Last-write-wins conflict resolution uses timestamps; skew can discard the newer write. Lease-based locks assume the holder’s clock expires before another node takes over; skew can allow two holders. JWT validation, cache TTLs, and rate-limit windows also depend on time. Logs merged from several hosts can appear out of order, complicating incident analysis.
Mitigations
Keep NTP healthy and monitor offset. Avoid wall-clock ordering for correctness; use version numbers, vector clocks, or a centralized sequencer. For leases, use fencing tokens so stale holders cannot commit, and include a safety margin larger than the maximum expected skew. Systems needing tight bounds can use specialized clocks with uncertainty intervals and wait out the uncertainty before committing.
Trade-offs and example
Logical clocks capture causality but grow with the number of actors and require pruning. Centralized timestamps simplify ordering but add a dependency and latency. Bounded-uncertainty clocks require infrastructure and waiting.
Example: two replicas update the same record. Replica A’s clock is 300 ms fast, so its older write carries a later timestamp and wins under last-write-wins, silently losing the newer value. A version vector or a server-assigned sequence would preserve both updates and expose the conflict instead of hiding it.