Continuous Profiling
Continuous profiling samples CPU, memory, and lock behavior of production code at low overhead, correlating profiles with deploys and traffic to catch regressions no benchmark reproduces.
Definition
Continuous profiling runs statistical profilers — sampling CPU stacks, allocation sites, lock contention — permanently in production at low, bounded overhead. Profiles aggregate fleet-wide, tagged by service version, and correlate with deployments and traffic patterns. The result: always-current flamegraphs answering what code consumed resources during any window, including the windows nobody was watching.
Context and scope
Delivered by Pyroscope, Parca, and platform profilers. Complements tracing (where time goes across requests) with code-level attribution (where time goes inside processes).
Common misconceptions
- Profiling is for incident response — continuous profiling is for before the incident: regressions, leaks, and contention appear as profile diffs between deploys
- Sampling profilers distort production — statistical sampling at 100Hz-class rates costs single-digit percent CPU; instrumenting profilers are the ones that distort
- Profiles replace traces — traces attribute latency across services; profiles attribute consumption inside processes; production debugging needs both
Anti-patterns
- Enabling profiles without deploy correlation and finding hot stacks without knowing which version introduced them
- Dropping labels from profiles to save cardinality and losing the ability to compare versions or regions
- Profiling only in emergencies, when the interesting window has already rolled out of memory
Related concepts
Related examples
Authoritative source
What it is
A profiler attached during an incident sees only the incident. Continuous profiling keeps a low-resolution record of everything, so the question “when did this allocation site appear and what changed then” has an answer.
How it works
Sampling profilers interrupt threads and record stack snapshots at fixed intervals; eBPF variants capture kernel and user stacks without application instrumentation. Samples aggregate into per-service, per-version profiles with time windows. Diffing profiles across deploys surfaces new hot paths; allocation profiles surface leak candidates before OOMs.
When to adopt
Production services with recurring performance mysteries, fleets where deploys are frequent enough that attributing a regression to a change is the daily question, and any platform expected to debug in production rather than reproduce locally. It also pays off on slow leaks that only long-term profiles reveal.
Pitfalls
Profile cardinality mirrors metrics cardinality: per-instance labels multiply storage; aggregate to useful dimensions and keep raw samples bounded. Overhead budgets differ by language runtime — verify on the slowest paths before fleet-wide rollout. Sensitive data in stacks — secrets on the call path — needs redaction policies like every other telemetry.