Go — CPU Throttling in Containers: GOMAXPROCS Mismatch

Diagnose latency spikes from Go running with host-sized GOMAXPROCS inside CPU-limited containers, read throttling metrics, and align the value with the quota.

advanceddebugginggocontainersperformance

Symptoms

Containers show periodic latency spikes despite low average CPU, with scheduling delays in the tens of milliseconds
runtime.GOMAXPROCS(0) reports the host core count (for example 64) while the container CPU limit is 2
cgroup throttling counters (nr_throttled, throttled_time) climb steadily under load

Possible causes

The Go runtime reads GOMAXPROCS from the host CPU count, ignoring cgroup CPU quotas
A limit of 2 CPUs with GOMAXPROCS 64 yields 32 OS threads competing for 2 cores
The CFS quota period refills every 100ms, so bursts exhaust it and everything in the period stalls
GC worker threads scale with GOMAXPROCS, amplifying runnable-thread count during collections

Diagnosis steps

Compare runtime view with the quota

go run -e 2>/dev/null; kubectl exec pod -- sh -c "cat /sys/fs/cgroup/cpu.max 2>/dev/null || cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpu.cfs_period_us"

Interpretation

cpu.max of 200000/100000 means 2 CPUs; runtime.GOMAXPROCS(0) printing the node core count is the mismatch in one measurement.

Read the throttling evidence

kubectl exec pod -- sh -c "cat /sys/fs/cgroup/cpu.stat"

Interpretation

nr_throttled and throttled_usec increasing under load means the kernel is freezing the process after quota exhaustion — the direct cause of the latency spikes.

Verify the fix changes the runtime

GOMAXPROCS=2 go run ./app # or automaxprocs import

Interpretation

Latency spikes disappearing with an aligned GOMAXPROCS confirms quota mismatch as the mechanism.

Root cause

The Go runtime sizes GOMAXPROCS and its scheduler from machine CPU count, while containers enforce a cgroup CPU quota. With the mismatch, many runnable OS threads compete for a fraction of the cores; the CFS scheduler exhausts the quota early in each period and throttles the whole process, producing periodic multi-millisecond stalls.

Fix

Align the value with the quota: set GOMAXPROCS explicitly to the container limit in deployments, or import go.uber.org/automaxprocs to read cgroup limits at startup. Keep GC pacing in mind: fewer Ps reduce GC worker parallelism, so re-tune GOGC if collection latency changes.

Risk note

Hardcoding GOMAXPROCS in the binary breaks when the deployment changes its limit. Prefer automaxprocs or inject the value from the orchestrator.

Verify the fix

Under sustained load, assert throttling counters stay near zero, p99 latency loses its periodic spikes, and runtime.GOMAXPROCS(0) reports the quota-aligned value.

Symptom walkthrough

The latency graph saws: low average with periodic spikes synchronized to 100ms boundaries.

$ cat /sys/fs/cgroup/cpu.max
200000 100000        <- quota 2 CPUs per 100ms period

$ kubectl exec pod -- go-env GOMAXPROCS   # or print runtime.GOMAXPROCS(0)
64                    <- the host core count

The two numbers are the disease: 64 scheduler threads (plus GC workers) competing for 2 CPUs of quota. The kernel meter burns through the quota early in each period, then freezes the process until refill — the spike. The receipt is in cpu.stat:

$ cat /sys/fs/cgroup/cpu.stat
nr_throttled 18402
throttled_usec 7439121067   <- 7.4 seconds spent frozen

Fix in practice

Two standard options. Explicit and auditable — set it from the deployment environment:

env:
  - name: GOMAXPROCS
    value: "2"   # matches the resources.limits.cpu you declared

Or self-adjusting — import the library and forget the knob:

import _ "go.uber.org/automaxprocs" // reads cgroup quota at startup

func main() {
    // runtime.GOMAXPROCS now matches the container limit
}

After aligning, watch the second-order effect: GC runs with fewer dedicated workers, so collections may take slightly longer while overall p99 improves. Re-check GOGC on latency-critical services rather than assuming defaults still fit.