Redis — Periodic Latency Spikes from Fork and Swap

Track down Redis latency spikes caused by background save forks and OS swapping using LATENCY HISTORY, measure fork rates, and fix memory overcommit and THP.

advanceddebuggingredislatencylinux

Symptoms

redis-cli --latency shows periodic spikes of tens to hundreds of milliseconds every save interval
LATENCY HISTORY shows fork events correlated with RDB save or AOF rewrite times
The host shows swap activity (si/so in vmstat) synchronized with Redis pauses

Possible causes

BGSAVE forks the whole dataset: with huge RSS, the fork itself plus copy-on-write faults stall the main thread
VM overcommit settings refuse or delay the fork, or the fork succeeds but copy-on-write explodes under write load
Transparent huge pages amplify copy-on-write granularity from 4KB to 2MB
The instance shares the host with memory-hungry neighbors, pushing pages to swap

Diagnosis steps

Correlate spikes with fork events

redis-cli latency history fork && redis-cli info persistence | grep -E "rdb_last_save_time|rdb_last_bgsave_status|latest_fork_usec"

Interpretation

fork events in latency history plus latest_fork_usec in the hundreds of milliseconds identify fork cost as the spike source.

Check memory and swap pressure

vmstat 1 10; cat /proc/$(pgrep redis-server)/smaps_rollup | grep -E "Swap|AnonHugePages"

Interpretation

Nonzero si/so means swapping; AnonHugePages confirms THP is inflating copy-on-write cost.

Verify overcommit and THP settings

cat /proc/sys/vm/overcommit_memory; cat /sys/kernel/mm/transparent_hugepage/enabled

Interpretation

overcommit_memory=0 lets the allocator refuse forks under pressure; THP enabled means 2MB copy granularity.

Root cause

Persistence forks duplicate the page tables of the whole dataset. With a large RSS the fork takes visible time, and every write afterward triggers copy-on-write page faults whose cost scales with page granularity. Combined with host swap pressure, the main thread stalls in periodic spikes aligned with save or rewrite events.

Fix

Set vm.overcommit_memory=1 for Redis hosts, disable transparent huge pages (madvise or never), shrink instance memory to leave fork headroom, tune save frequency or prefer AOF with no-fsync-on-rewrite adjustments, and keep Redis hosts free of memory-hungry neighbors.

Risk note

Disabling persistence entirely removes the spikes by removing the safety net. Tune persistence instead of deleting it, unless the data is truly disposable.

Verify the fix

After tuning, latest_fork_usec stays under a few hundred microseconds per GB, latency history shows no fork events, and --latency is flat across save cycles.

Symptom walkthrough

The latency monitor separates real causes from folklore:

redis-cli latency history fork
1) 1) "1729584000"
   2) (integer) 412         <- 412 ms fork stall

Then connect the numbers: latest_fork_usec shows the same magnitude, rdb_last_save_time aligns with the spike timestamps, and vmstat shows si/so activity in the same window. The three-part mechanism: fork duplicates page tables for the whole RSS; copy-on-write then faults per written page; and swap makes any stall worse by pulling pages back from disk while clients wait.

Fix in practice

Host-level settings first, because they change the physics:

sysctl vm.overcommit_memory=1       # let the fork succeed
echo never > /sys/kernel/mm/transparent_hugepage/enabled   # 4KB COW granularity

Instance-level sizing and cadence second:

# redis.conf — shrink memory so fork has headroom:
maxmemory 6gb            # on a host with 16gb, leaving room to COW

# and spread the save pressure:
save 900 1               # instead of frequent saves under write load

For write-heavy datasets where COW dominates, AOF with a reasonable everysec policy often beats frequent RDB snapshots, and a replica with persistence off can serve reads while a dedicated node persists.